{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0_xya1nyDHfY",
   "metadata": {
    "id": "0_xya1nyDHfY"
   },
   "source": [
    "<table style=\"width:100%\">\n",
    "<tr>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<font size=\"2\">\n",
    "Supplementary code for the <a href=\"http://mng.bz/orYv\">Build a Large Language Model From Scratch</a> book by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
    "<br>Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
    "</font>\n",
    "</td>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<a href=\"http://mng.bz/orYv\"><img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/cover-small.webp\" width=\"100px\"></a>\n",
    "</td>\n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "l62zIRRSBy_R",
   "metadata": {
    "id": "l62zIRRSBy_R"
   },
   "source": [
    "# Converting Llama 2 to Llama 3.2 From Scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aFmxTQbwCUMl",
   "metadata": {
    "id": "aFmxTQbwCUMl"
   },
   "source": [
    "- This is a follow-up notebook to [Converting a From-Scratch GPT Architecture to Llama 2](./converting-gpt-to-llama2.ipynb), converting Meta AI's Llama 2 architecture model step by step to Llama 3, Llama 3.1, and Llama 3.2\n",
    "- The explanations are purposefully kept minimal in this notebook so as not to bloat it unnecessarily and focus on the main code\n",
    "- For more information about the architectures, please see the Llama 2 and Llama 3 papers\n",
    " - [Llama 2: Open Foundation and Fine-Tuned Chat Models (2023)](https://arxiv.org/abs/2307.09288)\n",
    " - [The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ohhMKUWvGm9z",
   "metadata": {
    "id": "ohhMKUWvGm9z"
   },
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gpt-to-llama/gpt2-to-llama2-llama3.webp?1\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ws0wsUzwLH2k",
   "metadata": {
    "id": "ws0wsUzwLH2k"
   },
   "outputs": [],
   "source": [
    "# pip install -r requirements-extra.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "JBpQwU89ETA1",
   "metadata": {
    "id": "JBpQwU89ETA1"
   },
   "source": [
    "- Packages that are being used in this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "34a9a440-84c2-42cc-808b-38677cb6af8a",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "34a9a440-84c2-42cc-808b-38677cb6af8a",
    "outputId": "e3d3d4b6-ee63-4e28-d794-e8b0bdd931fd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "blobfile version: 3.1.0\n",
      "huggingface_hub version: 0.34.4\n",
      "tiktoken version: 0.11.0\n",
      "torch version: 2.8.0\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "\n",
    "pkgs = [\n",
    "    \"blobfile\",         # to download pretrained weights\n",
    "    \"huggingface_hub\",  # to download pretrained weights\n",
    "    \"matplotlib\",       # to visualize RoPE with different base frequencies\n",
    "    \"tiktoken\",         # to implement the tokenizer\n",
    "    \"torch\",            # to implement the model\n",
    "]\n",
    "for p in pkgs:\n",
    "    print(f\"{p} version: {version(p)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "UJJneXpTEg4W",
   "metadata": {
    "id": "UJJneXpTEg4W"
   },
   "source": [
    "&nbsp;\n",
    "# 1. Convert the Llama model implementation step by step"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "v1zpfX2GHBKa",
   "metadata": {
    "id": "v1zpfX2GHBKa"
   },
   "source": [
    "- If you are new to implementing LLM architectures, I recommend starting with [chapter 4](../../ch04/01_main-chapter-code/ch04.ipynb), which walks you through the implementation of the original GPT architecture step by step\n",
    "- The [Converting a From-Scratch GPT Architecture to Llama 2](./converting-gpt-to-llama2.ipynb) then implements the Llama-specific components, such as RMSNorm layers, SiLU and SwiGLU activations, RoPE (rotary position embeddings), and the SentencePiece tokenizer\n",
    "- This notebook takes the Llama 2 architecture and transforms it into Llama 3 architecture by\n",
    "    1. modifying the rotary embeddings\n",
    "    2. implementing grouped-query attention\n",
    "    3. and using a customized version of the GPT-4 tokenizer\n",
    "- Later, we then load the original Llama 3 weights shared by Meta AI into the architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c14b9121-abe1-4a46-99b8-acdef71e5b41",
   "metadata": {
    "id": "c14b9121-abe1-4a46-99b8-acdef71e5b41"
   },
   "source": [
    "&nbsp;\n",
    "## 1.1 Reusing Llama 2 components"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dgDhJGJ6xR4e",
   "metadata": {
    "id": "dgDhJGJ6xR4e"
   },
   "source": [
    "- Llama 2 is actually quite similar to Llama 3, as mentioned above and illustrated in the figure at the top of this notebook\n",
    "- This means that we can import several building blocks from the [Llama 2 notebook](./converting-gpt-to-llama2.ipynb) using the following code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a5bc3948-231b-4f1f-8d41-24ad0b7643d0",
   "metadata": {
    "id": "a5bc3948-231b-4f1f-8d41-24ad0b7643d0"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import io\n",
    "import nbformat\n",
    "import types\n",
    "\n",
    "def import_from_notebook():\n",
    "    def import_definitions_from_notebook(fullname, names):\n",
    "        current_dir = os.getcwd()\n",
    "        path = os.path.join(current_dir, fullname + \".ipynb\")\n",
    "        path = os.path.normpath(path)\n",
    "\n",
    "        # Load the notebook\n",
    "        if not os.path.exists(path):\n",
    "            raise FileNotFoundError(f\"Notebook file not found at: {path}\")\n",
    "\n",
    "        with io.open(path, \"r\", encoding=\"utf-8\") as f:\n",
    "            nb = nbformat.read(f, as_version=4)\n",
    "\n",
    "        # Create a module to store the imported functions and classes\n",
    "        mod = types.ModuleType(fullname)\n",
    "        sys.modules[fullname] = mod\n",
    "\n",
    "        # Go through the notebook cells and only execute function or class definitions\n",
    "        for cell in nb.cells:\n",
    "            if cell.cell_type == \"code\":\n",
    "                cell_code = cell.source\n",
    "                for name in names:\n",
    "                    # Check for function or class definitions\n",
    "                    if f\"def {name}\" in cell_code or f\"class {name}\" in cell_code:\n",
    "                        exec(cell_code, mod.__dict__)\n",
    "        return mod\n",
    "\n",
    "    fullname = \"converting-gpt-to-llama2\"\n",
    "    names = [\"precompute_rope_params\", \"compute_rope\", \"SiLU\", \"FeedForward\", \"RMSNorm\", \"MultiHeadAttention\"]\n",
    "\n",
    "    return import_definitions_from_notebook(fullname, names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d546032d-fce4-47cf-8d0e-682b78b21c61",
   "metadata": {
    "id": "d546032d-fce4-47cf-8d0e-682b78b21c61"
   },
   "outputs": [],
   "source": [
    "imported_module = import_from_notebook()\n",
    "\n",
    "# We need to redefine precompute_rope_params\n",
    "# precompute_rope_params = getattr(imported_module, \"precompute_rope_params\", None)\n",
    "compute_rope = getattr(imported_module, \"compute_rope\", None)\n",
    "SiLU = getattr(imported_module, \"SiLU\", None)\n",
    "FeedForward = getattr(imported_module, \"FeedForward\", None)\n",
    "RMSNorm = getattr(imported_module, \"RMSNorm\", None)\n",
    "\n",
    "# MultiHeadAttention only for comparison purposes\n",
    "MultiHeadAttention = getattr(imported_module, \"MultiHeadAttention\", None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "979c7b6d-1370-4da1-8bfb-a2b27537bf2f",
   "metadata": {
    "id": "979c7b6d-1370-4da1-8bfb-a2b27537bf2f"
   },
   "source": [
    "&nbsp;\n",
    "## 1.2 Modified RoPE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "m9_oDcHCx8VI",
   "metadata": {
    "id": "m9_oDcHCx8VI"
   },
   "source": [
    "- Llama 3 uses rotary position embeddings (RoPE) similar to Llama 2 (for a detailed explanation, please see the [RoPE paper](https://arxiv.org/abs/2104.09864))\n",
    "- There are some subtle differences in the RoPE settings, though\n",
    " - Llama 3 now supports up to 8,192 tokens, twice as many as Llama 2 (4,096)\n",
    " - The base value for the so-called RoPE $\\theta$ (see equation below) was increased from 10,000 (Llama 2) to 500,000 (Llama 3) in the following equation (adapted from the [RoPE paper](https://arxiv.org/abs/2104.09864))\n",
    "\n",
    "$$\\Theta = \\left\\{\\theta_i = \\text{base}^{\\frac{-2(i-1)}{d}}, i \\in \\left[1, 2, ..., d/2\\right]\\right\\}$$\n",
    "\n",
    "- These $\\theta$ values are a set of predefined parameters that are used to determine the rotational angles in the rotary matrix, where $d$ is the dimensionality of the embedding space\n",
    "- Increasing the base from 10,000 to 500,000 makes the frequencies decay faster across the dimensions, which means that the higher-dimensional components are associated with smaller rotation angles than before, so the overall frequency range becomes more compressed rather than expanded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ec8ded9f-3709-475a-87ff-0faee176207e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnYAAAHWCAYAAAD6oMSKAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAcqBJREFUeJzt3Qd4U+XbBvA76d60lC723nuDCCiCqIgD5QMVREVBVBQ3CqioOBEHiqAo+lfBCSigoLJkT2WWvUoHpdCW7ibnu543TUgnHWkzev+u65jk5OTk7Wlqbt6p0zRNAxERERE5Pb29C0BEREREtsFgR0REROQiGOyIiIiIXASDHREREZGLYLAjIiIichEMdkREREQugsGOiIiIyEUw2BERERG5CHd7F8CRGI1GnD17FgEBAdDpdPYuDhERERFkLYnU1FRERUVBry+5To7BzoqEurp169q7GERERESFnD59GnXq1EFJGOysSE2d+cIFBgbauzhERERESElJURVP5pxSEgY7K+bmVwl1DHZERETkSErTTYyDJ4iIiIhcBIMdERERkYtgsCMiIiJyEexjR0RUTRkMBuTk5Ni7GETVnoeHB9zc3GxyLgY7IqJqOCdWXFwcLl68aO+iEFGeGjVqICIiosLz6DLYERFVM+ZQFxYWBl9fX07ITmTnf2ilp6cjISFBPY6MjKzQ+RjsiIiqWfOrOdTVrFnT3sUhIgA+Pj7qVsKd/G1WpFmWgyeIiKoRc586qakjIsdh/pusaL9XBjsiomqIza9Ervk3yWBHRERE5CIY7IiIyOH169cPjz/+uL2LQaWQnZ2NJk2aYOPGjaU6/ssvv1QjQp3Vc889h0cffRSOwiGD3bp16zBkyBBERUWpqsnFixdf8TVr1qxBp06d4OXlpT5Q8kEhIiJyZI899hg6d+6svrs6dOhQ5DH//fcf+vTpA29vb7UQ/FtvvXXF8546dQo33nij6rclnfGffvpp5Obmlvl7c/bs2WjQoIF67+7du2Pr1q1XfO85c+agYcOG6NWrl2Vfab/LHc2aNWswdOhQNVLVz89P/Y6++eabfMc89dRTWLBgAY4dOwZH4JDBLi0tDe3bt1cfqNI4fvy4+gD3798fu3fvVv+qe+CBB/DHH39UelmJiIgq4r777sPw4cOLfC4lJQUDBw5E/fr1sWPHDrz99tt46aWXMHfu3BJHPst3otScSa2ZhA4JbVOnTi3T9+aiRYswadIkTJs2DTt37lTfy4MGDbJMy1Hc1B0fffQR7r//frgCuX7t2rXDTz/9pAL2mDFjMGrUKPz222+WY0JDQ9V1+eSTT+AQNAcnRfzll19KPOaZZ57RWrdunW/f8OHDtUGDBpXpvZKTk9X7yS0RkSvKyMjQ9u/fr26dSd++fbUJEyaoLTAwUKtZs6b24osvakaj0XLMV199pXXu3Fnz9/fXwsPDtREjRmjx8fGW55OSkrSRI0dqoaGhmre3t9akSRNt/vz5ludPnTql3XHHHVpQUJAWHBys3Xzzzdrx48er5OebNm2a1r59+0L7P/74Y1WWrKwsy75nn31Wa968ebHnWr58uabX67W4uDjLvk8++URdN/N5SvO92a1bN3W9zQwGgxYVFaXNmDGj2Pfetm2beu+UlJRSf5d/8cUX6pqbHTlyRF37sLAwzc/PT+vSpYu2atWqfK+pX7++Nn36dO2ee+5Rx9SrV09bsmSJlpCQoF4r+9q2bavKY5aYmKj93//9n/oZfHx8tDZt2mjffvutVlY33HCDNmbMmHz7FixYoNWpU0errL/NsuQTh6yxK6tNmzZhwIAB+fZJepb9JcnKylL/GrLeKtN/Zy5ixvID+N/mk5X6PkREZZocNTvXLpvp+770pObJ3d1dNQe+//77mDlzJj777DPL8zJNxPTp0/Hvv/+qZr8TJ07g3nvvtTw/ZcoU7N+/HytWrMCBAwdUDYvUtphfK98bAQEBWL9+PTZs2AB/f39cf/31quarOHJMSdu4ceNQEfI9dvXVV8PT09OyT8oZHR2NCxcuFPuatm3bIjw8PN9r5Dtu3759pfrelJ9Zagitj9Hr9epxSd+tcu2aNWumrmN5Xbp0CTfccAP++usv7Nq1S/0OpHuWNC9be++999C7d291jNQ+3nPPPao27e6771Y1jI0bN1aPzZ+zzMxM1ey9bNky7N27Fw8++KB6TWmal60lJycjJCQk375u3brhzJkz6jNnb+6uMou69QdYyGP5EGdkZFgm/itoxowZePnll6uolEDMsf3w2vA5ULMG0OPdKntfIqLiZOQY0Gqqfbqt7H9lEHw9S/81JP3L5Mtc+ms1b94ce/bsUY/Hjh1radI0a9SoET744AN07dpVBQUJWRIMOnbsiC5duqhjpO+YdbOj0WhUQdE87cQXX3yhOvVLPytpDi2KNGOWJDAwEBX9fpP+atbM33fyXHBwcKm/E83PleZ7U0KjNOkWdczBgweLLe/JkydV//iKkCZf2cwkrP/yyy9YunQpHnnkEct+CX8PPfSQui/NzBLUu3btijvuuEPte/bZZ9GzZ0/Ex8erpbpq166t+sOZyYAHaXr+/vvvVTArDTl227Zt+PTTT/PtN//M8vNbf67swSWCXXk9//zzqv+AmXyg5X8claWJdwoGe/yIkymyXAiDHRFRWfTo0SPfXF/ypf3uu++qACIz9UsNk/Q/kxo7CSYS1IQEulatWmH8+PG4/fbbVW2OBLVbbrnF0sFfXnPkyJFCNU1Sy3P06NFiyySDDugyCYUy0KIiJIjL71Fq1mJjY9WgDzlvwRo76ftmZg6gbdu2LbRP+gRKsJPPyeuvv67CWUxMjKqVlJa70k7WvXr1atXHbt68eWjdunW+58wVSLI0mL25RLCTX5gkcmvyWP6lVFxtnZCRQLJVlch6pv8BRGiJSE7LRpDf5ap1IiJ78PFwUzVn9npvWw66k6ZE2WTUYq1atVQQkMfmptTBgwerGpXly5dj1apVuPbaazFhwgS88847KkxIM13BEY9CzlUcqQksiTQLyihRW3+/mZ8r7jUFmxcLvuZK35sSlGUr6pji3ldI07bUpFaE1KrJ70d+LxKcpTzDhg0r1CTu4eFhuW8O/B5F7DMHfBl4Ik34s2bNUgFQRrnKoJGSmtrN1q5dq5qDpYZYmncLSkpKuuJnpaq4RLCTf7XJH6o1+VDIfkfiH1oPRujgpcvBwdMn0L5FM3sXiYiqOfnyK0tzqD1t2bIl3+PNmzejadOmKoBI8+D58+fxxhtvWFpetm/fXugc8sU7evRotckUIjINiAQImfZDmmNlapCyNJ9WdlOsfI+98MILqg+gObTI95s0RRfVDGt+zWuvvWZZd9T8GimL1FyW5ntT+vRJ0JV+blKzaQ5I8ti6ObQgaeqWJlHp11belRSkf6P0jbz11lvVYwndtui7tmHDBjV1iYRt889z6NAhyzUpjjTF33TTTXjzzTdVv7yiSJ89+f0UrMmzB4ccPCG/RPljMf/ByLBsuW+uhpUmVOvELJ1TZf6YZ555Rv1xf/zxx6qq9YknnoBDcfdEst70hxh/uviqfSIiKky+A6T7jAwc+O677/Dhhx9i4sSJ6rl69eqpMCL75PtA+mNJ3yxr0g9ryZIlqslVBhHIlBUtW7ZUz911112qtkm++GUAgHzvyBe6zDMnneKLIzVKJW3mYFUcKYt8v0mfN2luNH/3mWuRRo4cqX4umT5EyizhU2qdrLsRFSTNzBJWZGCANDFLP7IXX3xR1U6aW6lK870p7yHNjjJoRQabSFO21IxKc2RxZPoU+Q43D9KwZv4ut97kfAVJWP/555/V81J+uQbmWreKaNq0qQqvMoWJ/DzSP69gjWRRza8yMEM+B9KML78n2cw1dGbymZF/KJTUSlhlNAe0evVqNay34DZ69Gj1vNzK0PeCr+nQoYPm6empNWrUSA2fLquqmO7k9Js9NW1aoPb9V7Mr7T2IiFxxupOHH35YGzdunJq2Q6YAmTx5cr7pTmTqigYNGmheXl5az549taVLl6r/p+/atUs9L9NjtGzZUk11ERISog0dOlQ7duyY5fWxsbHaqFGj1HQocg75Lhk7dmylfifIz1XU9531NCv//vuvdtVVV6ky1a5dW3vjjTeK/M60fs2JEye0wYMHq59Vfp4nn3xSy8nJKfP35ocffqimEpFjZPqTzZs3X/FnuvPOO7Xnnnsu376ifkbZ1q9fX2i6E/k5+vfvr8pet25d7aOPPlLXaeLEifmmO3nvvfdKnFLl+PHj+X7/58+fV79zmQ5HplKR6XLk9y37iiN5o6hyF8wgMv3Md999pznCdCc6+Y+9w6WjkMETQUFBaihzRavPi3Nyzh2oH7cS3wSPx10T36iU9yAiKo4MBpCaExlpWdFO7uQYZPSuDAqQqVys+5jZi0zke91116lBJ1fqg+gKVqxYgSeffFL93DIdT2X8bZYlnzhkU6wr8wipp27dUmPsXRQiInIB0ldOgp0jhDrzaFXpjyYhpTpIS0tT4boioc6WHKMU1YjPVRNwza4WiM2qiaHZBvh42m5UGBERVT8//PADHI31xNCubtiwYXAkrLGrYsFRjXDBpz4yNC8cPXfJ3sUhIiIiF8JgZwdNwkx9DhjsiIiIyJYY7KpabjYe0n7AG+5zcSw2/3BpIiIioopgsKtqbh7od+4b/J/7GpyPtf9iwUREROQ6GOyqmk6HbD/TYsEZiSftXRoiIiJyIQx2dqAPNi13o0uJQY6h4rNpExEREQkGOzvwCjEFu3AtESfPp9u7OEREDq9fv35qwXZyDbLmrCznZjAYXP73n52djQYNGhS5dnFlYLCzA12QKdjV1iXiSEKqvYtDRER2otPpCm0LFy7Md4ysWdupUye1zqusP/vll18WOs/s2bNVeJAVC7p3746tW7eWav67Fi1aqNe0bdtWTXRsTRamkvV1IyMj1RqoAwYMwOHDh/MdI2umyjq7shpCjRo11Jq2slbslcgatbJ+rZubaS5X+Znk9c7ok08+UZMyyzWQrWfPnmo1CjNZ6/epp57Cs88+WyXlYbCzh6A66iZSdx5HEjjlCRFRdSarFsTGxlq2W265xfKcrN4gi9D3798fu3fvVrVWDzzwAP744w/LMYsWLcKkSZMwbdo07Ny5E+3bt8egQYOQkJBQ7Htu3LgRI0aMUEFs165d6j1l27t3r+WYt956Cx988AHmzJmDLVu2wM/PT51Xlr4yk1C3b98+rFq1Cr/99hvWrVuHBx98sMSf959//lHLjd1+++1wBXXq1MEbb7yBHTt2qFq5a665BkOHDlXXxfo6yc9tva/SVGjFWhdTlkV2K+TI35o2LVCLntJSm/jdzsp9LyKiUi407shk0fUJEyaoLTAwUKtZs6ZaxN1oNFqO+eqrr7TOnTurRd7Dw8O1ESNGaPHx8Zbnk5KStJEjR2qhoaGat7e31qRJE23+/PmW50+dOqXdcccdakH64OBg7eabb1YLyVemggvXF/TMM89orVu3zrdv+PDh2qBBgyyPu3Xrpq6LmcFg0KKiorQZM2YUe94777xTu/HGG/Pt6969u/bQQw+p+3JdIyIitLffftvy/MWLFzUvLy/LYvfyOZLyb9u2zXLMihUrNJ1Op8XExBT73lLWYcOG5dv3xRdfqOte0u9/4sSJpf5dr169WpXt999/1zp06KB+3/3791fHLF++XGvRooUWEBCgXpeWlpav/L1791ZlCQkJUdfoyJEjWlnJ5+ezzz7Lt0/eXz6z5fnbLEs+YY2dPdTthvXXr8DN2a/iCCcpJiIqlQULFqj1OKWZ8f3338fMmTPx2WefWZ7PycnB9OnT8e+//2Lx4sU4ceJEvqWtpkyZgv3796tmsgMHDqgmtNDQUMtrpTYqICAA69evx4YNG9QC9tdff73qI1UcOaakbdy4cVf8uSZMmKDK0a1bN8yfP181gZpt2rRJNYFak3LKfiFlk5oi62P0er16bD6mKFc6r9QUxsXF5TtGFqGXZl7zMXIrzaddunSxHCPHy/tLDV9x5Ppav6Y8rvS7NnvppZfw0UcfqRrK06dP484778SsWbPw7bffYtmyZVi5ciU+/PDDfOu+Su2n1LxJP0D5WW699VYYjaUb6Ch9BqUpXc4jTbLW5PcrP3tl41qx9uDph6jG7ZCJCziakAajUYNer7N3qYioOstOK/45nRvg4V3KY/WAh8+Vj/X0K3MR69ati/fee0/1Q2vevDn27NmjHo8dO1Y9f99991mObdSokWpG7Nq1q+rzJSHr1KlT6NixoyVUSJ806+ZM+fKWoCjnNzeRSnCRPm4DBw4sskzSPFoS6XNVkldeeUU13fn6+qqQ8fDDD6vyPvbYY+p5CVfh4eH5XiOPU1JSkJGRgQsXLqgwUdQxBw8eLPZ9izuv7Dc/b95X0jFhYWH5npfgHRISYjmmKCdPnkRUlGnar/K60u/a7NVXX0Xv3r3VfWl2fv7551UzsLzGvM7r6tWrLf3fCjYPS9CuVauW+gdBmzZtUBz5LEqQk2Zqef9ffvkFrVq1yneM/Mzys1c2Bjs7qR/iCw83HTJyDIi5mIG6Ib72LhIRVWevl/BF23QgcJfVQvNvNwFyihnRX/8qYMyyy49ntQXSzxc+7qXkMhexR48eltAl5Iv03XffVcFGOuFLzZXU0EgtjgQecy2LBDr5kh0/frz64pZ+aBLUpE9Zr1691DHymiNHjqgaO2vyRS1BoDgymKEipBbRTEKn1PS8/fbblmDniiSQyoCNirjS79pMBjVYh1IJ0OZQZ95nPdBEBofIgBGpcUxMTMx33pKCnfxDQ0J+cnIyfvzxR4wePRpr167NVxYZgJKeXvkzYbAp1k7cd3+ND3zno4PuCJtjiYgqSAKRNCVKDdk333yDbdu2qVoTYW5KHTx4sKoxeeKJJ3D27Flce+21arSikJqezp07qy9n6+3QoUMYOXJkpTbFWpOmzjNnziArK0s9joiIQHx8fL5j5LH8nBIUpAlXQm1Rx8hri1Pcec2vMd9e6ZiCAzRyc3PVSNmS3lvKLGGsMn/XZh4eHpb78o8C68fmfdbNrEOGDFHlnzdvngp35iblkprjzSNfJeTLZ2jGjBlqAIt0F7Am55Xav8rGGjt7OfQHBueswgZ9HRxNuIT+zfNXZxMRVanJZ0tuirX29JESji1QX/D4HthKwX5bmzdvRtOmTVWwkWbH8+fPq9GJ0mQripo3TL5YpTZFtj59+uDpp5/GO++8o6YTkeZYaVq8UvOpLZtiizpfcHCwmtrEXCtZcBoSGYFq7r8lgULChPQHM4+mlaAijx955JFi30deL8dYzw1nfd6GDRuqcCbHdOjQQe2T5l/5HUjNp/kcFy9eVLVnUgbx999/q/eXgFocqZmUps3yKu3vuqzknNHR0SrUyWdDyEjW8pBrYA7nZjLiWH72ysZgZ+cpT6I45QkROYKy9HmrrGOvQJrDpGP7Qw89pJpTpdO7NMWKevXqqZAj+6SWTL5EpXO9NWlikwDSunVr9aUr03PIJLnm6SikCVSmqZB+bzKFhdTu/fzzz2rONXls66bYX3/9VdWASROzNE1KsHr99dcttYhCfhbp/C9lkH5lEpy+//571fHfTK6JBFXpOygd9GVwgNRqjRkzptj3njhxIvr27auun0ynIh3+JRzNnTvXUpMloU/6qEl4lqAnzcbST8wcIOXayeAS6eMoU6LIgAYJk//3f/9XYh86qW2TgTAFSZN6waAsAdf8OzIrze+6PCRQ16xZU10DmbtPPm/PPffcFV8n/fakNljKlZqaqgZmSL9M6ylphAycsEU5r6jMY3hdWJVNdyL+maWmPPnlxeu12z/eUPnvR0Tk5NOdPPzww9q4cePUdCcyncTkyZPzTXfy7bffag0aNFBTcvTs2VNbunSp+n/6rl271PPTp0/XWrZsqfn4+KipLIYOHaodO3bM8vrY2Fht1KhRajoUOUejRo20sWPHVtp3gkytIVNxyJQdfn5+Wvv27bU5c+ao6UqsydQdcpynp6cqk0wNUtCHH36o1atXTx0j059s3rw53/OjR49W19Da999/rzVr1ky9RqZUWbZsWb7n5dpOmTJFTSci1+Paa6/VoqOj8x1z/vx5NWWI/AzyexkzZoyWmppa4s8tr5HpRw4ePGjZJz+T/K4Kbo0bNy5yupMr/a5X5013cuHChRKnVJk2bZq67marVq1SnxE5b7t27bQ1a9ZccUqa++67T6tfv766jrVq1VLXaeXKlfmO2bhxo1ajRg0tPT290qc70cl/Kj8+OgepZpbh3NL5sazV52W29yfgx/uw1dgcY91exe6p1+XrFExEVBlkMIBMZSE1MBXtwE7OQ2rnZJJjGXDgCKQJXL5zP/30U1QHw4cPV/3uJk+eXK6/zbLkEw6esJe8ZcWkKTY5IweJl0rumElERFQeEgZkZK91E6+9vfDCC6hfv36p54dzZtnZ2WrJNhm0UxXYx87OfewidEnQw6j62dUKMHWWJSIishWp6ZGRto5E5gcsqfbKlXh6eqp1casKa+zsxT8c0LvDHUbUwkVOeUJEREQVxmBnL3o34JFteKvzGsQjRE15QkRERFQRDHb2FNIIDSJM6xRyyhMiIiKqKAY7O2scZlrT7nBCqr2LQkTVCCdEIHLNv0kGO3s6thZtd7yAe9xWIj4lCymZOfYuERG5OPOSSlWxZiURlZ75b7LgsmdlxVGx9nT+CDz/+wYDPLvi64yBqp9dx3rB9i4VEbkwWX5LRiSa1/iURdE5hyaRfWvqJNTJ36T8bcrfaEUw2DnAXHb13ZMs/ewY7IiospkXaC+4gDsR2Y+EOvPfZkUw2NlTUG11E2Y8p2455QkRVQWpoZO1MGXBe1nfk4jsS5pfK1pTZ8Zg5wCTFPsaUuCDTByJZ7AjoqojXyS2+jIhIsfAwRP25B0EeAValhZjjR0RERFVBIOdg9TaSbA7nZSOzByDvUtERERETorBzt4CTf3s6numwKgBxxPT7F0iIiIiclIMdvZ2yyfA5FjsD7tJPeQKFERERFReDHb25l8L8PRFk7wVKBjsiIiIqLwY7BwEgx0RERFVFIOdvV08DSx5BEOOTVcPGeyIiIiovDiPnb1pBmDX14hw8wIwTA2eyDUY4e7GzE1ERERlw/RgbwFRMg88dIYsRHlcQrbBiNMXMuxdKiIiInJCDHb25u4J+Ieru11rpKtbNscSERFReTDYOdAkxW0DUtXt4QTTLREREVFZMNg5ULBr6p2sblljR0REROXBYOdAwa6e23l1Gx3HGjsiIiIqOwY7Bwp2tdxMfewOx19CjsFo50IRERGRs2GwcwSdRqllxXzvnAt/L3c1MvboOTbHEhERUdkw2DkCTz+1rJher0PLyAC160Bsir1LRURERE6Gwc7BtIwMVLf7zzLYERERUdkw2DmKZU8BX92CLjXS1MP9rLEjIiKiMmKwcxTHVquttU+SenggNhWaptm7VEREROREGOwcbGRsXbck6HVAUlo24lOy7F0qIiIiciIMdo4i0BTsPC+dReNa/uo+B1AQERFRWTDYOViNHVLOXB5AwWBHREREZcBg52jBLvkMWkUx2BEREVHZMdg5YLAz19gd4JQnREREVAYMdo4W7HLS0Sov2B0/n4b07Fz7louIiIichru9C0B5QhqpZcVkBYpaAEL9vZB4KQsH41LRqV6wvUtHREREToA1do5C76ZCnZm5nx1HxhIREVFpMdg5KHNzLJcWIyIiotJiU6wj2ToPOLgM6Hg3Wkb2VLtYY0dERESlxRo7R5J42LS0WPxetM5ripU+dgYjlxYjIiKiK2OwcyRBtU23F0+jQU0/eLnrkZ5twMnzafYuGRERETkBBjtHUqOe6fbiSbi76dEiIkA9PBCbat9yERERkVNw6GA3e/ZsNGjQAN7e3ujevTu2bt1a4vGzZs1C8+bN4ePjg7p16+KJJ55AZmYmnEZIY9Nt0jF1c3lpsWR7loqIiIichMMGu0WLFmHSpEmYNm0adu7cifbt22PQoEFISEgo8vhvv/0Wzz33nDr+wIED+Pzzz9U5Jk+eDKeay06knwcyLlpNecIaOyIiInLiYDdz5kyMHTsWY8aMQatWrTBnzhz4+vpi/vz5RR6/ceNG9O7dGyNHjlS1fAMHDsSIESOuWMvnULz8Af9w0/2ko5dr7DjlCRERETlrsMvOzsaOHTswYMAAyz69Xq8eb9q0qcjX9OrVS73GHOSOHTuG5cuX44Ybbij2fbKyspCSkpJvc4jmWN9QVWNn7mMXl5KJpLRse5eMiIiIHJxDzmOXmJgIg8GA8PC82qs88vjgwYNFvkZq6uR1V111FTRNQ25uLsaNG1diU+yMGTPw8ssvw6GMWgy4e6m7Euvq1/TFyfPpaj673k1C7V06IiIicmAOWWNXHmvWrMHrr7+Ojz/+WPXJ+/nnn7Fs2TJMnz692Nc8//zzSE5OtmynT5+G3eWFOrOWEVxajIiIiJy4xi40NBRubm6Ij4/Pt18eR0REFPmaKVOm4J577sEDDzygHrdt2xZpaWl48MEH8cILL6im3IK8vLzU5shkAMXv++LYz46IiIics8bO09MTnTt3xl9//WXZZzQa1eOePU1LbRWUnp5eKLxJOBTSNOs0UmKBr28FPu1bYMoTBjsiIiJywho7IVOdjB49Gl26dEG3bt3UHHVSAyejZMWoUaNQu3Zt1U9ODBkyRI2k7dixo5rz7siRI6oWT/abA55T8AoAjv5tup9xwTLlyZGES8jKNcDL3Yl+FiIiIqpSDhvshg8fjnPnzmHq1KmIi4tDhw4d8Pvvv1sGVJw6dSpfDd2LL74InU6nbmNiYlCrVi0V6l577TU4FfOUJ5fi1UTFUVGdEOjtjpTMXBXuWkcF2buERERE5KB0mlO1U1Yume4kKChIDaQIDDTVlNnF/MHAqY3A7Z8DbYfh/+ZuwuZjSXh7WDvc0aWu/cpFREREDp1PHLKPXbVnXoGi0NJi7GdHRERExWOwc0Q184Ld+aPqplVesOOUJ0RERFQSBjsnqLEzD6CQKU/Yck5ERETFYbBzRLKsmE8I4G0aKNEkzB/uep0aQHE2OdPepSMiIiIH5bCjYqu1iLbAs8ctD2WKEwl3B+NSVa1d7Ro+di0eEREROSbW2Dkina7QLvazIyIioithsHMS1v3siIiIiIrCYOeoNn0MfNARWP9uvilPDsQx2BEREVHRGOwcVW6GaVTsuUP5gt3J8+lIzcyxc+GIiIjIETHYOfyUJ6a57EL8PBER6K3uR8el2rNkRERE5KAY7Bx5yhOruezy9bPjAAoiIiIqAoOdowppaLpNPw9kXMw3MpYDKIiIiKgoDHaOyisA8A/PV2vXprYp2P17JtmeJSMiIiIHxWDnRM2x7evWULfRcSlIz861Z8mIiIjIATHYObLI9kBUJ8DN0/QwyAfhgV4wasAe1toRERFRAVxSzJENfqPQrg51a+CPffH498xFdG9U0y7FIiIiIsfEGjsn06FusLrdfdo0oIKIiIjIjMHOGRgN+WrsxO5TDHZERESUH4OdI8vNBj7qCrwWAWSa+tS1qxMEvQ44m5yJhJRMe5eQiIiIHAiDnSNz9zQFOkM2cN60AoWflzuahQeo+7vYHEtERERWGOyccAWK9nXymmMZ7IiIiMgKg53TrBl7Odh1qMd+dkRERFQYg52zLC1mHezyBlD8d+YiDDKpHRERERGDnROomdcUm9fHTkgfO19PN6RlG3Ak4ZL9ykZEREQOhcHOCZti3fQ6tK0dpO7vPn3BXiUjIiIiB8Ng5wzBLqId0LAPYMgp3M/uNJcWIyIiIhMuKebovAKAcesL7e5onqiYI2OJiIgoD2vsnHxpsei4FKRn59q7OEREROQAGOychdEIZF0eKBER5I2IQG/IoNg9Z9gcS0RERAx2zmH7fNOyYsufyrfbsm4sm2OJiIiIwc5J+AQDhqx8U56I9gx2REREZIXBzqmmPMkf7FhjR0RERNYY7Jwp2KWfBzIuh7h2dYKg1wGxyZmIT8m0X/mIiIjIITDYOcuUJ35hhSYq9vNyV6tQiF1cN5aIiKjaY7BztqXFrIKdYHMsERERmTHYOYuQKwU7Li1GRERU3XHlCWdRrweQlQLUbJJvt3lpMZnLzmDU1DqyREREVD0x2DmLTveYtgKahgXAz9MNadkGHEm4hOYRpj53REREVP2wKdbJSQ1d2zpB6j6bY4mIiKo3BjtnW1YsOQbIzS5y3VgOoCAiIqreGOycyQcdgPdaAQn78u3uUNdUY8cpT4iIiKo3BjtnEhBpui2wtJi5xu5QfCrSsnLtUTIiIiJyAAx2LjCXXUSQNyICvWHUgD0xyfYpGxEREdkdg50zLi1WoMZOcKJiIiIiYrBzJqHNTLfnDhZ6yjyf3W72syMiIqq2GOycSXjry8HOaMj3FGvsiIiIiMHOmQQ3ANx9gNxMIOl4vqfa1g6CLDoRl5KJuORMuxWRiIiI7IfBzpno3YDO9wK9Hwc8vPM95efljmbhplUnWGtHRERUPZU62B09WrjDPtnB4DeA614GguoUeqqjuZ8dgx0REVG1VOq1YseNG4cjR44gIiIC7dq1y7cFBZkmyCX76lgvGN9tPY3tJ5LsXRQiIiJy5Bq7VatW4fjx4xgyZAgSEhIQExODV199FSEhIWjSpEnllpLyu5QAnNpcaHf3hiHq9t8zF5GRnX9wBREREbm+UtfYmX3//ffYvXu35fHKlSvxzTff2LpcVFKoe6cpoNMDk88CHj6Wp+qF+CIyyBuxyZnYeeoCejcJtWtRiYiIyMEHT3h7e2P//v2WxwMHDsTevXttXS4qjl8twLcmoBkLzWen0+kstXZbjp23UwGJiIjIaWrsPv/8cwwfPhz9+vVDhw4dsGfPHhUoqIrItQ5rBZxYD8TvB6I65nu6R6OaWLz7LDYfYz87IiKi6qbUNXbHjpnWJ23dujV27NiBPn364MSJE6hfvz5WrFhRmWWk4iYqTrhcc2rWvVFNy8jYzBz2syMiIqpOSl1j9+KLL6pw16lTJ9X8OnjwYNx5552VWzoqmtTYifh9hZ5qUNMXYQFeSEjNUv3sejVmPzsiIqLqotTB7ttvv4Wmaaq2TgZMfPjhh+px//79VdDr1q0bm2QdoMZOfgfSHLv037PYciyJwY6IiKgaKdPgCQkNXbp0weTJk/HXX39h6dKlaN++Pb766it079698kpJ+dVqYbq9FA+kFR4k0b2RaQDFZg6gICIiqlbKPHjCmr+/P26++Wa1URXy8gf6PgcERgFuHoWelho7sSuvn523h5sdCklEREQOHey2bNmimmQ3btyI2NhY+Pr6omXLlqq/3YgRI7gCRVXq/3yxTzUK9UOovxcSL2Xh39MXLQMqiIiIyLWVuin2pptuwhdffKH600kTrIyI3blzJ15++WVkZWVh2LBhaj/Zn5rPztIcy2lPiIiIqgudJiMgSuHixYuoUaNGhY9xZCkpKarWMTk5GYGBgXBoOZlA7L+mfnatCjeFf735JKYs3otejWvi27E97FJEIiIiqtp8Uuoau5ICmzTRXukYsrELJ4D5A4HF4wGjsdDTPfJWoNhx8gKycjmfHRERUXVQ5iXFinLHHXegMsyePRsNGjRQy5jJqNutW7descZwwoQJiIyMhJeXF5o1a4bly5fDJdVsDOg9gOxLQPKpQk83CfNHTT9PZOUa8d+ZZLsUkYiIiBx08ERxkxFLS25Sku37cS1atAiTJk3CnDlzVKibNWsWBg0ahOjoaISFhRU6Pjs7G9ddd5167scff0Tt2rVx8uRJ161FlNGwtZoD8XuBhANAcIMi+9kt3xOHzUfPo2sDUw0eERERua5SB7s///wTX3/9tZripGCwW7dunc0LNnPmTIwdOxZjxoxRjyXgLVu2DPPnz8dzzz1X6HjZLwFTRux6eJimAJHaPpdfgUKCnaxA0XxwkdOeSLDbcjwJj9qlgEREROSQwa5fv34ICAjA1VdfXei5du3a2bRQUvsmK1w8//zlKT30ej0GDBiATZs2FfkaGZHbs2dP1RS7ZMkS1KpVCyNHjsSzzz4LN7ei53GT0byyWXdOdCrhrYA9Ra9AIbo3rGnpZ5eda4Snu01a3omIiMhBlfqb/ueffy4y1IlVq1bZskxITEyEwWBAeHh4vv3yOC4ursjXyDq20gQrr5N+dVOmTMG7776LV199tdj3mTFjhhplYt7q1q0LpxKWt7RYfNHBrmmYP4J9PZCRY8CemItVWzYiIiKqci5ThWM0GlX/urlz56Jz584YPnw4XnjhBdWEWxypEZShw+bt9OnTcLoaO3H+MJCbXehpvV5nqbXjfHZERESuzyGDXWhoqGo+jY+Pz7dfHkdERBT5GhkJK6NgrZtdZVUMqeGTpt2iyMhZmQ/GenMqgbWBG94B7lksoyWKPITrxhIREVUf5Q52xTWJ2oKnp6eqdfvrr7/y1cjJY+lHV5TevXvjyJEj6jizQ4cOqcAn53NJEua6jQUa9ilyzVjrdWOln12OofB8d0REROQ6yh3sZGmxyiRTncybNw8LFizAgQMHMH78eKSlpVlGyY4aNSrf4Ap5XkbFTpw4UQU6GUH7+uuvq8EU1Vnz8ADU8PVAerb0s+N8dkRERK6s1KNiCyrlSmTlJn3kzp07h6lTp6rawQ4dOuD333+3DKg4deqUGilrJgMf/vjjDzzxxBNqlK7MYychT0bFurRL54AjfwLGHKDTqCL72ckcdqv2x2PLsSR0qhdsl2ISERGRA60VW5CEp//++w+uxKnWijU7uRH4YjAQVBd4Ym+Rh3z+z3FM/20/+jWvhS/HdKvyIhIREZGDrRVLDiqspek2+TSQWXRTa/e8dWO3HU9CLvvZERERuSwGO2fnE2waHStkabEitIwMRKC3O9KyDdh31skmYSYiIqLKD3bFreZAdlpaTMjSYkVw0+vQLa/WjtOeEBERua5yB7tdu3bZtiRU8YmKi1lazHraE1k3loiIiFwTm2JdwRWWFhPmFSikn53BWLkjmomIiMg+GOxcqsZun8xDU+QhraICEeDljtSsXOxnPzsiIqLqHezee+89dbtv3z4YDIbKLBOVVWgz4K4fgfGbij1E+tl1zetnt+U4+9kRERFV6wmKZYJgMXnyZBw8eBA+Pj5o3bo12rZtizZt2uCmm26qzHJSSdy9gKbXXfGwHo1C8PfBBKw/nIgH+jSqkqIRERGRA9bY9e/fX90uWbIE0dHR+Oeff/DYY48hNDQUf/75Z2WWkWykb7Mwy8jYzBzWuhIREVXbYHfdddfhs88+U8t8CX9/f3Tv3h333Xcf7rzzTjz88MP48ssvK7OsVJKEg8DfrwKbPyn2kGbh/ogM8kZWrhGbOO0JERFR9Q12ixcvRlpammpybdiwITp16oRWrVqhcePGmDt3LkaPHo177723cktLxUs6Cqx7G9j9TbGH6HQ6tayYWBttCuhERERUDfvY+fn5YeLEiWrLzs7G+fPn4e3tjeBgLirvUJMUnzsEGHIBN/dim2O/23oaa6ITAORNk0JERETVd7oTT09PREZGMtQ5khr1AQ8/wJBlqr0rRu8mNeHhpsOJ8+k4kZhWpUUkIiIiBwp2GRkZiImJKbRfpkAhO9PrgbAWJS4tJgK8PdClvmnaE1OtHREREVW7YPfjjz+iadOmuPHGG9GuXTts2bLF8tw999xTWeWj8jTHJhwo8TBzP7vV7GdHRERUPYPdq6++ih07dmD37t344osvcP/99+Pbb79Vz2nFrHZAVSy8jek27r8SD+vXnNOeEBERVevBEzk5OQgPD1f3O3fujHXr1uHWW2/FkSNH1GhLcgC1O5luEw+VeJhMexIV5I2zyZlq2pP+eUGPiIiIqkmNXVhYGP7773JNUEhICFatWoUDBw7k2092FNkBmLAVeGRHiYdJEO+bF+Y47QkREVE1DHZff/21CncFR8d+9913WLt2bWWUjcrK3ROo1dw0kOIKzP3sOICCiIioGga7OnXqICIiwvI4Li7Ocr937962LxlVqt5NQi3TnhzntCdERETVdx47MXDgQNuWhGwj8TDw4/3AD2NKPMzfy53TnhAREbmYcgc7joR1UDo9sPdH4OAyIDe7xEP7tzA3x7KfHRERUbUOdhwJ66BCGgE+waYVKOL3lngopz0hIiJyLeUOduSgJHDX7my6H1Py6NimYaZpT7JyjWraEyIiInJuDHauqHYX0+2Z7aWe9mTNQfazIyIiqrbBzs3NzbYlIdupkxfsYkoOdqK/edqTQ+xnR0REVG2D3a5du2xbErIdc1Ps+SNAxoUSD+2VN+3JSU57QkRE5PQq3BS7fv163H333ejZsydiYmIskxn/888/tigflYdvCBDaDIhoC6TGX3Hak64NOO0JERERqnuw++mnnzBo0CD4+PioGrysrCy1Pzk5Ga+//rqtykjlMX4TMO4fIKxFGVahYHMsERFRtQ12r776KubMmYN58+bBw8Mj30oUO3futEX5qLzc3Et9aP+8ARQyMjYjm9OeEBERVctgFx0djauvvrrQ/qCgIFy8eLEipyZbyc2S2aRLPKRJmD9q1/BBdq5RzWlHRERE1TDYydqxR44cKbRf+tc1atSoIqemipIw98UNwIw6QNKxUkx7Ym6OZT87IiKiahnsxo4di4kTJ2LLli0qHJw9exbffPMNnnrqKYwfP952paTyTVQstXWG7CtOVCz6NeO0J0RERM6u9B2xivDcc8/BaDTi2muvRXp6umqW9fLyUsHu0UcftV0pqfzz2clcdjJRcbs7Szy0d4FpTxqG+lVZMYmIiMgBauyklu6FF15AUlIS9u7di82bN+PcuXOYPn26jYpHFVKnq+n2zLYrHurn5Y5uDU3TnqzmKhRERETVd0kxT09PtGrVCt26dYO/v78tTkm2nKg4bg+Qk1nq0bEr98dVdsmIiIjI0YJdRkaGaoI1O3nyJGbNmoU//vjDFmWjigpuAPjWBIw5pnB3Bde3iVC3W48n4VyqaU5CIiIiqibBbujQofjqq6/UfZneRGrs3n33Xdxyyy345JNPbFVGqsgAitqlXze2TrAv2tcJglED/tjHWjsiIqJqFexkEuI+ffqo+z/++KOa/kRq7STsffDBB7YqI1VE0+uAljcDwQ1LdfgNbSPV7Yq9sZVcMCIiInKoYCfNsAEBAer+ypUrcdttt0Gv16NHjx4q4JED6DYWGP410Pz6MgW7TUfP4/wlNscSERFVm2DXpEkTLF68GKdPn1b96gYOHKj2JyQkIDAw0FZlpCpUN8QXbWubm2Pj7V0cIiIiqqpgN3XqVDVnXYMGDVT/up49e1pq7zp27FiRU5OtV6FIOg6klK55lc2xRERE1TDYDRs2DKdOncL27dvzjYSVCYvfe+89W5SPbGHZk8AHHYAdX5Tq8BvamkbHbjx6Hklp2ZVcOCIiInKIlSeEDJiQCYqlli47+3IIiIuLQ4sWLSp6erKF8FalnqhY1K/ph9ZRgdh3NgWr9sdheNd6lVs+IiIisn+wO3bsGG699Vbs2bNHrUKhSZNf3ooUwmAw2KaUVDGWKU92AEYjoNeXqjlWgt2yPQx2RERE1aIpduLEiWjYsKEaLOHr64t9+/Zh3bp16NKlC9asWWO7UlLFhLcG3L2BzGQg6WipXjI4b7LijUcScTGdzbFEREQuH+w2bdqEV155BaGhoWqaE9muuuoqzJgxA4899pjtSkkV4+YBRHYw3T9z5YmKRaNa/mgREYBco4aV+zk6loiIyOWDnTS1muexk3B39uxZdb9+/fqIjo62TQnJNuqUfgUKsxvzRscu38PRsURERC4f7Nq0aYN///1X3e/evTveeustbNiwQdXiNWrUyFZlJFuo3blMNXZicF6w23AkEcnpOZVVMiIiInKEYPfiiy/CKJ3xARXmjh8/rpYYW758OZcUczT1egC9HgP6PlPqlzQJ80fz8ADkGDSsOsDmWCIiIken08xDWW1Epj4JDg62jIx1JikpKQgKCkJycjJXzsgz689DmPXnYVzTIgzz7+1q7+IQERFVOyllyCcVqrGzJvlQtpCQEKcMdVRyP7v1h88hJZPNsURERI6swsHu888/V33tvL291Sb3P/vsM9uUjmwrJxM4uhr4d1GpX9I0PABNw/xVc+yfHB1LRETk2mvFylx2Q4YMwQ8//KA2uf/EE0+o58jBxO0Bvr4FWPEMYDSUeRDF8j1xlVg4IiIismsfu1q1aqlBEiNGjMi3/7vvvsOjjz6KxMREOBOX72NnyAXeaghkpQAPrgGiOpbqZdFxqRg0ax083fXY8eIABHh7VHpRiYiIqIr72OXk5KhVJgrq3LkzcnNzK3Jqqgxu7kCDq0z3j5V+ZZBm4f5oVMsP2blG/HUgofLKR0RERBVSoWB3zz334JNPPim0f+7cubjrrrsqcmqqLI36lTnYyWAYTlZMRETk+NzL+oJJkybl+8KXgRIrV65Ejx491L4tW7bg1KlTGDVqlG1LSrbRsK/p9tRm02AKD+9SvWxwm0h8+PcRrDl0DpeycuHvVeaPDhEREVWyMn8779q1q1Czqzh69KhlaTHZ9u3bZ6syki3Vag74RwCX4oDTW4BGeUHvClpGBqBhqB+OJ6bhrwPxGNqhdqUXlYiIiCo52K1evbqsLyFHInMMSpj7bxFwcmOpg53Uzt7QNgKzVx/Fkt1nGeyIiIgckM0mKCYnctUTwLgNQN9ny/Sy2zrVUbdrohMQn5JZSYUjIiKi8mKwq47CWgIRbQB92X79jWv5o0v9YBg14OedMZVWPCIiIiofBjsqkzu6mGrtfth+Wi0hR0RERC4U7M6cOQOj0Vjovi3Mnj0bDRo0UEuVde/eHVu3bi3V6xYuXKj6hN1yyy02K4vLOb0N+PlBYPWMMr3sxnZR8PFww7HENOw4eaHSikdERER2CHatWrXCiRMnCt2vqEWLFqmpVaZNm4adO3eiffv2GDRoEBISSp4gV97/qaeeQp8+fWxSDpclo2JlAMXen8r0Mpnm5Ia8Oe1+2H6mkgpHREREdgl21s1xtmyamzlzJsaOHYsxY8aowDhnzhz4+vpi/vz5xb7GYDCoiZFffvllNGrUyGZlcUmyAoVOD5w/DCSXrb/cnXnNsb/9dxbp2VxhhIiIyFE4ZB+77Oxs7NixAwMGDLDs0+v16vGmTZuKfd0rr7yCsLAw3H///aV6n6ysLLX+mvVWbfgEX14r9vjaMr20W8MQNKjpi7RsA5bviauc8hEREZFrBLvExERV+xYeHp5vvzyOiys6SPzzzz/4/PPPMW/evFK/z4wZM9Siuuatbt26qJarUJRheTEh/ReHdTbV2n2//XRllIyIiIhcJdiVVWpqqlq3VkKdrHpRWs8//zySk5Mt2+nTp6vpurFrpR29TC+9vXMdNdfx1uNJOJGYVjnlIyIiojJxyAU/JZy5ubkhPj4+3355HBERUeh4Wc5MBk0MGTLEss88Otfd3R3R0dFo3Lhxodd5eXmprdqq2x1w9zYNpDgXDYS1KPVLI4N8cHXTWlh76Bx+3HEGTw1qXqlFJSIiIietsfP09FRr0P7111/5gpo87tmzZ6HjW7RogT179mD37t2W7eabb0b//v3V/WrXxFpaHt5A/V5AeFsg40K557STYGeQWYuJiIjIrhyyxk7IVCejR49Gly5d0K1bN8yaNQtpaWlqlKwYNWoUateurfrJyTx3bdq0yff6GjVqqNuC+6mAkd8Dbh7leul1rcJRw9cDcSmZWH/4HPo1D7N58YiIiKgKg93kyZMREhJS6H5FDR8+HOfOncPUqVPVgIkOHTrg999/twyoOHXqlBopSxVUzlAnvNzdcEuH2vhy4wn8sOMMgx0REZGd6TSuC2Uh053I6FgZSBEYGIhqJTsdMOYA3kFletnemGTc9OE/8HTTY8vkaxHs51lpRSQiIqqOUsqQT1jlRaZlxd6sD2yZW+aXtqkdhFaRgcg2GLFkd9kmOiYiIiLbYrAjwL8WYMgu83x2BQdRSHMsERER2Q+DHQGN+ptuz2wFsss+J530s5Om2H1nU7DvbLLty0dERERVG+zee+89dbtv3z61agQ5kZBGQFBdU63dqeKXbCuO9KuTEbLih+2stSMiInL6YCejVs0jY1u1aqUe33XXXXjjjTfw22+/2eptqDLIEhLlXF7MbFhec+zi3THIymWwJyIicupgJ5MBiyVLlqiVHmTt1scee0ytIvHnn3/a6m2oKpYXKwdZhSIi0BsX03Owan/+FUOIiIjIQeexk3VZX3rpJSxfvhyJiYlq+G2zZs3Qu3dv3H777WoVCOHv74/u3burjZxAw6tNt3H/AWnnAb+aZXq5m16HYZ3r4KPVR/DVppO4qV1U5ZSTiIiIbFdjJys+/PDDDxg5ciRee+01PProo/j777/x9ddfo3Xr1mopr5gYTnvhdALCgS73A4NmAOWc+PmuHvXgrtdh6/EkNb8dEREROfgExX5+fqqZtWPHjpZ9AQEB+Pfff+Hm5qbC3rJly9QxDRs2hDOp1hMU28hj3+3C0n/P4raOtTFzuKnfJRERETnoBMWypFd6enqRz9WvXx9z587F+PHjMXHixLKemlzA/VeZwvyv/51FQkqmvYtDRERUrZQ52D3yyCO47777VA1dce6++27VPEtO6NI5YPsXwNnd5Xp5+7o10KV+MHIMmuprR0RERA4c7CZNmoQhQ4agU6dOuP766zFnzhwYjUboZMqMPAsXLlSjYckJ/fUS8NvjwK6vK1xr982Wk8jM4dQnREREVaVcveTfeecdbNy4UfWte/LJJ5GRkYH27dujUaNGqFmzJqZPn463337b9qWlytdyqOn2wG+A0ViuUwxsHYE6wT64kJ6Dn3dyIA0REZHDTndiJtOYyOjY7Oxs7Ny5E4cOHVKd+6Sm7pprrkFYWJhtS0pVo1FfwCsQuBRnWmKsXo8yn0KmPrm3VwO8uuwA5m84jhHd6uar0SUiIiIHC3Zmnp6e6NGjh9rIBbh7Ac2uB/Z8D+xfWq5gJ+7sWhfvrTqEIwmXsO5wIvo2q2XzohIREVElrTxBLqTVzabbA78CZZsNxyLQ20OFO/H5P8dtWToiIiIqBoMdFdb4WsDDF0g+BZzdVe7TjOnVUC1Du+7QORyOT7VpEYmIiKgwBjsqzNMXaHodoHMzLTFWTvVq+uK6luHq/vwNJ2xYQCIiIioKgx0VbcDLwFOHgc73Vug05qlPft55Bklp2TYqHBERERWFwY6KFtIQ8KtZ4dN0axiCNrUDkZVrxLdbOGExERFRZWKwoyvLzSr3S2WaE3OtnaxEkZ1bvrnxiIiI6MoY7Kh456KBzwcB866p0GlubBuFsAAvJKRmYdmeszYrHhEREeXHYEfF8w8DYrYD8XuBxCPlPo2nux6jeta3TH2ilXMKFSIiIioZgx0VzycYaNjXdP/AkgqdamT3+vBy12NvTAq2Hk+yTfmIiIgoHwY7KlnLIaZbWYWiAkL8PHF75zrq/kery1/7R0RERMVjsKOStbgJ0OmB2N3AhYqNah3ftzHc9TqsP5yIbSdYa0dERGRrDHZUMv9aQL1el5cYq4C6Ib64o4tpmbGZKw/ZonRERERkhcGOyrB2bMWaY8Uj1zSBp5sem46dx8ajiRUvGxEREVkw2FHp+tk1Gwx0ua/Cp6pdwwf/181Uazdr1WGOkCUiIrIhBju6ssAoYORCoP3/2eR0D/droqZA2XoiCRuOnLfJOYmIiIjBjuwgIsgbd3Wvp+7PXBXNWjsiIiIbYbCj0ks6Dmz4AEireN+48f0aw9tDj52nLmLtoXM2KR4REVF1x2BHpff9KGDVFODgsgqfKizAG/f0MK1GMXPVIdbaERER2QCDHZVe61tMt/9+Z5PTPdS3MXw83PDfmWT8dSDBJuckIiKqzhjsqPTajwB0bsCpTUDCwQqfLtTfC6N7NVD3WWtHRERUcQx2VLbRsc2uN93fucAmp3zo6kbw83TD/tgU/LEv3ibnJCIiqq4Y7KhsOt97uTk2J7PCpwv288R9VzVU92f9eQhGI2vtiIiIyovBjsqmybVAYB0g40KFlxgze+CqRgjwcsfBuFSs2Btnk3MSERFVRwx2VDZ6N6DTKMDDF7hkmxAW5OuB+/tcrrUzsNaOiIioXBjsqOy6PwQ8GQ30etRmp5Tm2EBvdxxOuIQlu2Nsdl4iIqLqhMGOys6nBuAdaNNTBnp7YFy/xur+m78fRFpWrk3PT0REVB0w2FHFnN0F5GbZ5FT39W6IeiG+iE/Jwkerj9jknERERNUJgx2V33cjgLn9gIO/2eR03h5umHJTK3X/8/XHcTwxzSbnJSIiqi4Y7Kj8ItqZbnd8abNTDmgZhr7NaiHbYMT03/bb7LxERETVAYMdlV/HuwGdHji+Djh/1Can1Ol0mDqkFdz1Ovx9MAF/H+SkxURERKXFYEflV6Mu0OQ6m65EIRrX8rdMWvzKr/uRlWuw2bmJiIhcGYMd2WYlil3fALnZNjvto9c0Qa0AL5w4n475/5yw2XmJiIhcGYMdVUzTgUBAJJCeCEQvs9lpA7w98Nz1LdT9D/8+jLjkii9fRkRE5OoY7Khi3NyBjveY7u9fatNT39qxNjrWq4H0bAPeWHHApucmIiJyRQx2ZJvm2BELgdvm2fS0er0OL9/cGjodsHj3WWw7kWTT8xMREbkaBjuquKDaQPPBpto7G2tXpwaGd6mr7k9bso/ryBIREZWAwY5sy5Br2mzoqUHNEeDtjv2xKfhu6ymbnpuIiMiVMNiR7Wz6GJjVxmYrUZiF+nth0nXN1P13VkbjQprtRt8SERG5EgY7sp3080BqLLDxA0CzbZPp3T3qo1m4Py6m5+DVZRxIQUREVBQGO7Kd7uMAdx8gZgdw9G+bntrDTY/Xb22rBlL8tPMMVu3nihREREQFMdiR7fjXArqMMd1f97bNa+26NAjB2D6N1P3nf96DJDbJEhER5cNgR7bV6zHAzQs4tQk48Y/NTy997ZqE+SPxUhamLNlr8/MTERE5MwY7sq3ASKBT3oTF696y+em9Pdww8872cNPrsOy/WPz671mbvwcREZGzYrAj2+v9OKD3AI6vA84dqpS57Sb0a6zuS61dQiqXGyMiIhIMdmR7NeoCN74DjN8I1DJNU2Jrj1zTFK0iA9Uo2ck/74Fm4/58REREzojBjipvmbHw1pV2ek93PWYObw8PNx3+PJCAn3bGVNp7EREROQsGO6p8aYmVctoWEYF4fICpRvDlpftw9mJGpbwPERGRs2Cwo8ojzaO/TgTebQHE/lcpb/HQ1Y3QoW4NpGbl4tmf/mOTLBERVWsOHexmz56NBg0awNvbG927d8fWrVuLPXbevHno06cPgoOD1TZgwIASj6cqILMJZ10CjDmmee0qgbubHu/e2R5e7nqsP5yIb7ZwLVkiIqq+HDbYLVq0CJMmTcK0adOwc+dOtG/fHoMGDUJCQkKRx69ZswYjRozA6tWrsWnTJtStWxcDBw5ETAz7XtnV1U+Zbg8sBRIqZymwxrX88cz1LdT915cfwInEtEp5HyIiIken0xy07Upq6Lp27YqPPvpIPTYajSqsPfroo3juueeu+HqDwaBq7uT1o0aNKtV7pqSkICgoCMnJyQgMDKzwz0B5Ft1jCnZthgHDPq+UtzAaNYyYtxlbjiehZWQgfh7fCz6ebpXyXkRERFWpLPnEIWvssrOzsWPHDtWcaqbX69VjqY0rjfT0dOTk5CAkJKTYY7KystTFst6oElz9tOl2389A4pFKeQu9XodZ/9cBof6eOBCbghd+4RQoRERU/ThksEtMTFQ1buHh4fn2y+O4uLhSnePZZ59FVFRUvnBY0IwZM1QCNm9SI0iVILId0GwwoBmB9e9W3tsE+eDDEZ3UqhQ/74rB/zafrLT3IiIickQOGewq6o033sDChQvxyy+/qIEXxXn++edVtaZ5O336dJWWs1rpm1drF70cyEqttLfp2bgmnr2+ubr/ym/7sePkhUp7LyIiIkfjkMEuNDQUbm5uiI+Pz7dfHkdERJT42nfeeUcFu5UrV6Jdu3YlHuvl5aXaqq03qiS1OwO3fAI8tgvwCqjUtxrbpxFuaBuBHIOGh7/ZgXOpWZX6fkRERI7CIYOdp6cnOnfujL/++suyTwZPyOOePXsW+7q33noL06dPx++//44uXbpUUWmp1DqMBHyL7/NoKzqdDm8Na48mYf6IT8nCI9/uRK7BWOnvS0REZG8OGeyETHUic9MtWLAABw4cwPjx45GWloYxY8ao52WkqzSlmr355puYMmUK5s+fr+a+k754sl26dMmOPwUVSQY1RK8ADDmV9hb+Xu6Yc3dn+Hm6qZGyb/5+sNLei4iIyFE4bLAbPny4aladOnUqOnTogN27d6uaOPOAilOnTiE2NtZy/CeffKJG0w4bNgyRkZGWTc5BDubnscB3/wdsmVOpbyM1du/c0V7dn7f+OH7772ylvh8REZG9Oew8dvbAeeyqyK7/AUsmAJ7+wCPbgMCoSn27GSsO4NO1x+Dr6YbFE3qjWXjl9vEjIiKyJaefx45cXPuRQJ1uQPYlYOWLlf52Tw9sjp6NaiI924BxX+9ASmblNQETERHZE4MdVT29HrjxHUCnB/b+BBxfV6lvJ+vJfjiyIyKDvHEsMU2Fu6xcQ6W+JxERkT0w2JF9RLYHutxvur/86UodSCFC/b0wb1QXNZhi49HzmLToXxiM7IVARESuhcGO7OeaFwDfUODcwUofSCHa1A7Cp/d0gYebDsv2xOKVX/dx2TEiInIpDHZkPz7BwHUvAzWbAhElTyZtK1c1DcW7d3ZQ9xdsOomP1xytkvclIiKqCu5V8i5EJQ2kaHsn4O5ZZW95c/sonL+UhZd/3Y+3/4hGqL8nhnetV2XvT0REVFlYY0f2H0hhHeqMVbNCxJjeDfFwv8bq/vM/78Gq/fmXryMiInJGDHbkGGTwxIYPgM8HVPpACrOnBzXHHZ3rQMZQyLJj208kVcn7EhERVRYGO3IMMqfdhveBmB3A+plV8paypuyM29ri2hZhyMo14v4F23EoPrVK3puIiKgyMNiR4wykuH6G6f7aN4CTG6vkbWWOu49GdkKnejWQnJGD0fO34nRSepW8NxERka0x2JHjaHcn0H4EoBmBnx4A0qumadTH0w2fj+6q1paNTc7EnZ9uwrFzl6rkvYmIiGyJwY4cyw3vACGNgZQY03qyVTTPXLCfJ755oLtVuNuM6Dg2yxIRkXNhsCPH4uUP3PEF4OYJRC8HtnxaZW8dHuiNRQ/2QKvIQCReysLwuZuw50xylb0/ERFRRTHYkWMuNzbwVUDvUeVvXdPfC9+N7YEOdWvgYnoORs7bzNGyRETkNHQa11SySElJQVBQEJKTkxEYGGjv4lRv8rFMPAzUamaXt7+UlYv7vtyGrceT4OMhffC6oFeTULuUhYiIqreUMuQT1tiRY9Lp8oe6nMwqfXt/L3csGNMNfZqGIiPHgHu/3IbVBxOqtAxERERlxWBHji9uDzDnKmD3t1X6tjJa9rPRXXBdq3Bk5xrx4NfbsWJPbJWWgYiIqCwY7MjxHV4JnD8MLHsSOHeoSt/ay90NH9/VCUPaRyHHoGHCtzvx1aYTVVoGIiKi0mKwI8fX+3GgYV8gJx34cQyQk1Glb+/hpses4R0wvEtdtfzY1CX78MIve5BjqJp1bYmIiEqLwY4cn94NuG0u4BsKxO8FFo8HjFUbqtz0Orxxe1s8P7iF6v73zZZTuOfzLbiQll2l5SAiIioJgx05h4AIYNh80xQo+34Bfn+2yiYvtl5b9qG+jTHvni7w83TD5mNJGDp7Aw5zfVkiInIQDHbkPBr1BW6dY7q/dS6w4wu7FGNAq3D8/HBv1An2wamkdNz68UaOmCUiIofAYEfOpe0w4Po3gHq9gNa32a0YzSMCsGRCb3RrGGKa827BNsxbdwycFpKIiOyJExRb4QTFTsSQC7i527sUahqUqUv2YuG20+rxsM518OotbeDt4WbvohERkYvgBMXk+qxD3Za5wKktdimGp7seM25ri6k3tYJeB/y44wxu/ugfHIxLsUt5iIioemOwI+e2+ztgxdPAt3cCCQftUgQZVHHfVQ2x4L5uCPX3xKH4S7j5ow1YsPEEm2aJiKhKMdiRc2s1FKjTFci8CPzvNiA5xm5F6dO0FlZMvBr9mtdSTbTTlu7DAwu24/ylLLuViYiIqhcGO3Junr7AyO+B0GZASgzwv9uB9CS7FadWgBe+uLcrpg1pBU83Pf46mIDr31+PdYfO2a1MRERUfTDYkfPzDQHu/hkIiALOHQDmDwIumgYz2KtpdkzvhljySG80DfPHudQsjJq/Fa/+th9ZuQa7lYuIiFwfgx25hhp1gVGLgcDaQOIhU7jLumTXIrWMDMTSR67C3T3qqcef/XMct87eiL0xyXYtFxERuS4GO3IdtZoD968CarUEej0KePnbu0Tw8XTDq7e0xbxRXRDs64H9sSlq1Owrv+5X898RERHZEuexs8J57FxETgbg4XP5sSEHcPOAvSWkZqpA99t/sepxZJA3Xrq5NQa1jrB30YiIyIFxHjuq3qxDXcYFYG5/YNtnsLewAG98NLITvhzTFXVDfBCbnImHvt6hRs7GXMywd/GIiMgFMNiR689zF78HWPYk8PdrgANUUPdrHoaVj/fFw/0aw12vw58H4nHdzLVqSbJcg9HexSMiIifGplgrbIp1QfLxXvsmsGaG6XGn0cCNMx1iOTJxKD4VL/yyB9tOXFCPW0QE4IUbW6o58YiIiMqaTxjsrDDYubDt8021dpoRqNsduP0zoIZptKq9GY0afthxGjNWHMTF9By176omoXj2+hZoWyfI3sUjIiI7Y7ArJwY7F3dwOfDLQ0BWCuAdZJrYuF4POIqktGx89PcRfL35BHIMpj/LIe2j8NTAZqhf08/exSMiIjthsCsnBrtq4MIJ4Mf7gNQ4YNw/psmNHczppHTMXHUIi3fHqJZk6Yd3V/d6eOSapmplCyIiql5SGOzKh8GumpDpTy6eAmo2Nj2WP4HUWCAwCo5k39lkvPV7NNbmLUfm6+mGB/o0wn29G6CGr6e9i0dERFWEwa6cGOyqqV3/A5Y/DdzwNtDhLlkTDI5k49FEvLniIP49k2wJeP/XtR4e6NMQUTWspnYhIiKXxGBXTgx21ZB8/BfeBUQvMz1uM8wU8BysiVb+TFfsjcOHfx/BgdgUtU+aaId2qI1xfRuhaXiAvYtIRESVhMGunBjsqimjEdjwXt48dwbAJxgY8BLQcRSgd6ypHuXPdd3hRMxZcxSbjp237B/QMgzj+zVG5/qOFUiJiKjiGOzKicGumju9Dfj1MSBhv+lxVCfg5g+AiLZwRLtPX1QB74/9cZZ5l7vUD8bdPerj+jYR8PZws3cRiYjIBhjsyonBjtTAiq3zTBMaZ6UCY/8GaneCIzt67pJateLnnTHIzlu5ItjXA8M618GIbvXQqJa/vYtIREQVwGBXTgx2ZJEaDxz5E+h41+V9Z3YAUR0AvWPWhCWkZGLhttNYuPUUziZnWvb3alwTI7vXw8BWEfB0d6ymZSIiujIGu3JisKNiJR4GPu4JhLcC+r8INL3O4UbPmhmMGtZEJ+CbLaewOjrB0kwb6u+J2zvXwS0daquly3QOWn4iIsqPwa6cGOyoVKtWiMgOwNVPA81vcLgBFtZiLmZg0dZTqiYvITXLsr9ZuD9ubh+Fm9vXRr2avnYtIxERlYzBrpwY7KhEl84BG98Htn0O5KSb9oW1Bq5+Cmg11GGbaEWOwYi/DiTgl11nsPrgOUtfPNGhbg0M7RCFG9tFIizA267lJCKiwhjsyonBjkol7TyweTawZS6QnQp4BQFP7DGtP+sEkjNy8Me+OCzdfVZNfmzM+z+AXgd0b1gT17UKx4CW4azJIyJyEAx25cRgR2WSccEU7jx9gV6PXp4Tb9s8oPWtgH8YHF1CaiaW/ReLpf+exa5TF/M9J82117Y0hTyp1XOT5EdERFWOwa6cGOyowo7+DXx9K6B3B1rcCHQaDTTq79D98MxOnU/Hyv1x+PNAPLaduKAGYZjV9PPENS3C1NazcU2uVUtEVIUY7MqJwY4q7Pg64K9XgDPbLu+rUR/odA/Q4W4gMBLOIDk9B2sOJeDPAwlqhG1qZq7lORlM2yYqCL2a1ETvxqHo2iAEPp6O27+QiMjZMdiVE4Md2UzcXmDnAuDfRUBWsmmfzg14eBNQqzmciQy82HY8SYW89YfP4XDCpXzPe7rp0al+DRXypDavTe0grnpBRGRDDHblxGBHNpedDuxfYgp56UnAhC2X57/b+KFpwEXzGwG/mnAW8SmZatDFhiPnsfFIYr7JkM1Br03tQHSuH6y2TvWCERbI0bZEROXFYFdODHZUqTJTAO+8z1VuFvB2E9O8eFKT16A30PJmoMVNTtNcK+R/HyfOp+OfI4kq5G07kYTES9mFjqsb4oPO9YLVIIy2dYLQMjIQvp7udikzEZGzYbArJwY7qjKyDu3mOcCBpUDcf/mfq9US6Ho/0G0snI387+R0UgZ2nErC9hMXsOPkBUTHp1pWvzCTAbaNa/mrZtvWUYFoWzsIraICEeDtYa+iExE5LAa7cmKwI7tIOg4c+NUU8s5sl3hkWras79Om56UJd/PHQMO+QN1ugLsXnElqZg52n76ogt6emGTsjUnOtwqGtfo1fdE0LADNI/zRLDxAbY1q+cHLnX32iKj6SmGwKx8GO7I7CXEysjaiLVCzsWnfvl+AH+413XfzAiLbA3W6ALU7m25l1K2TrfuakJKJvWcl5KWosLcvJrlQXz0zmT+vQU1fNI8IQJNa/mhYyw8NQ/3RsKYfgnxZw0dEri+Fwa58GOzIIZ3aYpr0+NhaIC2h8PPDvgDa3Ga6n5YIGLKBgEinC3vnL2UhOi4Vh+JTER1/Sd3KZj3VSkEhfp5oGOpn2aTGr06wL+oG+6jndE52DYiIisJgV04MduTQ5E816ZipuTZmu+k2bg/wyDYgpKHpmPXvmubR864BhLUCwlvl3bYGwlo6zbJnZvK/p7iUTBySoBeXimOJl3DsXBpOnE9DfErRzblmvp5uqBPsg7oS9EIk8Pmgdg0fRAR5I6qGD0L9vbiaBhE5BQa7cmKwI6eTk2nqc2eumfrjBVN/PM1Y9PEPbzYFPHF6G3ApDghpBAQ3NC2N5kQuZeXiRGIajlttp5LSceZC+hVDn5BQFx7ghci8sBcZ6I3wQG+EBXqhlr9X3q03An3cWfNHRHbFYFdODHbkMmEv8RCQsB+I35d3u9/UjDs5FnDPWw5s8QRg9/8uvy4gCghuAATVMW19JgFeAabnDDmAm/P0Z8vMMSDmYgbOXMjA6aR0nL6QjjNJGYhNli1TzcVntWJaiTzd9ZagV9PPC6H+nqqZV7aa/p5qn/l+sK8nJ2cmIrvmE04kReRqPLyByHamreA8euZQJ4LrA1GdgKSjQGYykHrWtJn1e+7y/V8nmkbuSt+9gHDA37yFmW5b33p5tK78W9HONVwSrmQ6FdmKkmswqvn2ziZnIC45U4W92IsZarRuQmomzqnbLNW/LzvXqEKibKV7b70KeEE+Huq2hq9H3mbaF+jtgQBvdwSq++ZbD1UzyNG/RFRRDh3sZs+ejbfffhtxcXFo3749PvzwQ3Tr1q3Y43/44QdMmTIFJ06cQNOmTfHmm2/ihhtuqNIyEzks8+TIZn2fMW3m0bjnjwIXTwIpMabH1tOqJJ82TaYsW2J04XO3zhu8IX4eCxxeCfjWvLz5hAC+IYBPMNDrscsBU6Z6kcmafWqY+v+5e1dJKHR306vmV9muVPNnDnnnUjNVGExKM22Jl7Is98+nZeNCWjZyjRoyc4ymoFjMKN+SyKod/t7u8Pdyh5+XOwK83PM99vdyUxM7+xW89XRX6/X65m0+Hm7qsdzKz0pE1YfDBrtFixZh0qRJmDNnDrp3745Zs2Zh0KBBiI6ORlhYWKHjN27ciBEjRmDGjBm46aab8O233+KWW27Bzp070aZNG7v8DEROQ0KXbHW7Fv38iIVAcgyQGgtcSgAuxZv658n97LT8NYEyMldqAGWTwR4F9X788v2/pwN7f7r8WO8OePqbmoDl9oE/Aa+8Wrdd3wCxuwEPH8DDz9Qn0Pp+s+svh9GUWCAn3fTYXY7xNoVGvVuZa/5k4IVsVyK9WlKzcpGcnoML6dm4mHebnJGDC2mm+1IDmJKZg5SMHKTIfXWbYxn5m20wWsKirXi46SxBT34eb3e51cPLw/xYb7qVfe5u8HLXq+ZndV/tM92XfWpzk1sdPN1M++T8l/frVZBU+9StPNbBQ6+HngNViKqEw/axkzDXtWtXfPTRR+qx0WhE3bp18eijj+K556yaiPIMHz4caWlp+O233yz7evTogQ4dOqhwWBrsY0dkA1Lbl3bOdJt+3rRlyL5EICcDuGnm5WMXPwxELzeFwKIGfEw5D7jl/fvzhzHAvp+Lf99nT5pq/sTSR4GdXxU+RoKjzAX46I7LS7etn2kKl26epiAofQn1HqbHcv+Gt4GACNOxB5cDx9eazqPO5ZH/foe7AL9Q07Gx/5pGLZuf1+lNwVLddwPqdTfVYEoXxotnkJFwDBm5QHqOZtpyNaRna0jLMSBOH4mLBk9kZBuQm3ERuvQLSM8xquPSso1IyzFtcvy5HC8k5+hVi7gHcuGJHGjQwQiduhVG6GUabBjUbdXU6LnrdZaQJ7cqAOp18JAwKLduejWgRfbLY7kvAdFNf/mx9a3e8vjy83qd6T3kVioq5Tm3vPtyvOm++XmdZZ9kTst9OVbtu3weGTxjfk5nec50nFQwm54z7QMuP2d+Xm9+bd5zss98HnmJ+bjL+03HWvaZ71sfK48sz5teZ7q9fLwqTYHX5zuOg4KchtP3scvOzsaOHTvw/PPPW/bp9XoMGDAAmzZtKvI1sl9q+KxJDd/ixYuLfZ+srCy1WV84IrJR7V9p3PKx6VZSiCyzJlv2JSDrkunWHOpEq6GmSZuz0021ceZNPc4APKxq1SSUeQWa9htzLu835po264Eg0swcv7f4Mg589fL9UxuBLSX8Q7HpwMvBTkLg2jeKP/aBv4E6nU3F3fcT/FdNRdE9AgGM/hVoeLXp/tZ5wPKnij/v3d9DazoQWblGGHf+D74rHiv20M2d38XRsOtU83Hts79j0P7n88KfhEBZA0UHLe/Lf37IE1jjda2qVWyTsQ3Ppsy4fIz5VpPQqMOM3BH43tBfva6zLhqfer6njjFFC0Az6CRVqn0f5t6K/xmuU/tb6U7gE893LOUzB1Hz7fzcwZhvGKzuN9DF4iuPN4o8Tiw09Mdsw83qfgTO41vP1wr9/Objlxp64X3D7ep+DaTiR8+Xi7xecvxKY2e8nft/6rE3srDU88VC5zNbb2yLV3PvsTz+3fNZ+bGLtNXYAlNzx1geL/Z8EV6w+uxa2WNshGdyH7r8s3pORxDSijz2kFYHE3MesTz+wuNNhOsuFnnsaYRhQu4kSyic7TYTdXVFzJ0J4ByCMV4zfUdLQHwDH6AJTpt/xfmkwh9j9S9ZnpqqzUFL7ajl+csv0SELHrjfY4Zlz9OGz9DeeLDggRb3er5ruf9I7gJ0NRZYotHKOK8ZyNaZavUfyPkWvQyy0k/RnvCZjks601/kXdk/or9hQ7HHPu89Bdm+4Vj6yFVwBA4Z7BITE2EwGBAeHp5vvzw+eDDvF1yA9MMr6njZXxxptn355aL/gImoCsk3ifQBLNgP0FrrW0xbadz4rmkTRgOQm2kaLSy3hixLTZnS42GgxU2miZ1l9G++22xTH0Gzhv1MoVGel/NKaFT35TbXNH+gmYTQpoMuh0k5XpPXyJZ7uYlZyOtqNsk7xph/k33SjGwmtX7S/FzwODm36WKqL1o1Otej5Nq4Ho1qokfr+qYHe8OA/eaYZiXv4UNXNcBDHXuYHkSnAN8VMZgk74v3zaHN8XrnwcgxaDAc84H/wuL/0TyuZzhuatUDuQYNPuc8EbUyqdhjh7bwR91GrWAwaghI9UC9reeKPbZPhBvSouqrfo9Bme5odKj474L2QTnoH1ILBg0IyHVDk7NWg4gKOOnTAq18A2HUNHga3dFM+qQW44x7PUT4eqtjZWuRe7rYY+N1ofDW6U2jtTWgue4MfHRFN8knF/gnQFPdGdTUpRZ5bJYx/9d8U30M6ugSizzW3ZirrpdZA7ezaKYr+ucL0NKQln05ptb1PIsW+pNFHnteC1BdE8wi1bEnijw2XfNS/VrNQj1i0dztuOlBEe2L1n1Za3jEopn52CLEXkxHBkwtA/4e8SUfm3QJSXm12d7u8WjqXvyxcRdSkZqRN4OAA3DIptizZ8+idu3aqt9cz549LfufeeYZrF27Flu2bCn0Gk9PTyxYsED1szP7+OOPVXCLj48vdY2dNPeyKZaInJb1qGQJhRI8zeFPvhnlefN9CYjm/pESfKVJ3HyMujVevi8DYMxhVPpVploFJcvxeV8n/rUuh2epfb146vK3csFjpZlbRlebz5t4uPCx6j5MzeeBUXnHppum87lciPzXQc5bo97ln+3sruKvmby/eQm/3GzgzLaiLqzpxi8MqNXs8vU9ubHo9xe+oaZJws1k9ZjiyPWyHskuSwtad0/QNPUOkrs07yBokR1MNaSaBt3JDer3LEeYL6v5ymke/jBEdrY853Zmk7oe6jmrY9Wthy+yI039bOV495ht0EmtuDrWaDlePXbzQlbtnuo44Rm7A7psU7hUH7HLxVY17xm1e1keeyXshj5LPmvWZTDd0eCG9DpXWfZ7Jf4Ht8wLRV5eOeZSnastr/c+vw/uGYmFjjU/TI3qrbpCSOzxTjoIj3RTNigqBKVG9IQm/4iTYHfxEDzT8od962uRGt4dOk8fdKpn9Q9GG3P6ptjQ0FC4ubkVCmTyOCIir69LAbK/LMcLLy8vtRERuQzrflOqT18pB4zIABPZSsPT73IQuhIJg9bh5krnjepQymN9ix/sU5D8XPUvVxKUSIJug96lO1aubcM+KLVGfUt/rLnp3Yr8Zov8bTYpfGyxmpehDEFlOG/NMhwbXoZrFlWG5s06pfy9iXql/DyI+t3hTBxyHLzUvnXu3Bl//fWXZZ8MnpDH1jV41mS/9fFi1apVxR5PRERE5GocssZOyECI0aNHo0uXLmruOpnuREa9jhlj6lw6atQo1Vwr/eTExIkT0bdvX7z77ru48cYbsXDhQmzfvh1z5861809CREREVM2DnUxfcu7cOUydOlUNgJBpS37//XfLAIlTp06pkbJmvXr1UnPXvfjii5g8ebKaoFhGxHIOOyIiIqouHHLwhL1wHjsiIiJy5nzikH3siIiIiKjsGOyIiIiIXASDHREREZGLYLAjIiIichEMdkREREQugsGOiIiIyEUw2BERERG5CAY7IiIiIhfBYEdERETkIhjsiIiIiFwEgx0RERGRi3C3dwEciXnZXFmTjYiIiMgRmHOJOaeUhMHOSmpqqrqtW7euvYtCREREVCinBAUFoSQ6rTTxr5owGo04e/YsAgICoNPpyp2qJRiePn0agYGBNi+jK+O1qxhev4rh9asYXr/y47WrmOpw/TRNU6EuKioKen3JvehYY2dFLladOnVsci75cLnqB6yy8dpVDK9fxfD6VQyvX/nx2lWMq1+/oCvU1Jlx8AQRERGRi2CwIyIiInIRDHY25uXlhWnTpqlbKhteu4rh9asYXr+K4fUrP167iuH1y4+DJ4iIiIhcBGvsiIiIiFwEgx0RERGRi2CwIyIiInIRDHY2NHv2bDRo0ADe3t7o3r07tm7dau8iOaR169ZhyJAhaqJFmQh68eLF+Z6Xbp9Tp05FZGQkfHx8MGDAABw+fNhu5XUkM2bMQNeuXdUk2mFhYbjlllsQHR2d75jMzExMmDABNWvWhL+/P26//XbEx8fbrcyO5JNPPkG7du0s81317NkTK1assDzPa1c2b7zxhvobfvzxxy37eA2L99JLL6nrZb21aNHC8jyvXcliYmJw9913q+sj3w1t27bF9u3bLc/zu8OEwc5GFi1ahEmTJqmROTt37kT79u0xaNAgJCQk2LtoDictLU1dHwnCRXnrrbfwwQcfYM6cOdiyZQv8/PzUtZT/6VV3a9euVf/j37x5M1atWoWcnBwMHDhQXVOzJ554Ar/++it++OEHdbyspnLbbbfZtdyOQiYglzCyY8cO9YVwzTXXYOjQodi3b596nteu9LZt24ZPP/1UBWVrvIYla926NWJjYy3bP//8Y3mO1654Fy5cQO/eveHh4aH+MbZ//368++67CA4OthzD7448MiqWKq5bt27ahAkTLI8NBoMWFRWlzZgxw67lcnTyEfzll18sj41GoxYREaG9/fbbln0XL17UvLy8tO+++85OpXRcCQkJ6hquXbvWcq08PDy0H374wXLMgQMH1DGbNm2yY0kdV3BwsPbZZ5/x2pVBamqq1rRpU23VqlVa3759tYkTJ6r9vIYlmzZtmta+ffsin+O1K9mzzz6rXXXVVcU+z++Oy1hjZwPZ2dmqBkCqfa2XJ5PHmzZtsmvZnM3x48cRFxeX71rKMirStM1rWVhycrK6DQkJUbfyOZRaPOvrJ0099erV4/UrwGAwYOHChaq2U5pkee1KT2qNb7zxxnzXSvAaXpk0DUo3lEaNGuGuu+7CqVOn1H5eu5ItXboUXbp0wR133KG6oXTs2BHz5s2zPM/vjssY7GwgMTFRfUmEh4fn2y+P5YNGpWe+XryWV2Y0GlXfJmmeaNOmjdon18jT0xM1atTIdyyv32V79uxR/ZdkMtNx48bhl19+QatWrXjtSknCsHQ3kf6eBfEalkxCxpdffonff/9d9feUMNKnTx+1uDuvXcmOHTumrlnTpk3xxx9/YPz48XjsscewYMEC9Ty/Oy5zt7pPRE5Wa7J37958fXToypo3b47du3er2s4ff/wRo0ePVv2Z6MpOnz6NiRMnqv6dMkiMymbw4MGW+9I3UYJe/fr18f3336vO/lTyP2Slxu71119Xj6XGTv7/J/3p5G+YLmONnQ2EhobCzc2t0OgleRwREWG3cjkj8/XitSzZI488gt9++w2rV69WAwLM5BpJ14CLFy/mO57X7zKpFWnSpAk6d+6sap1kIM/777/Pa1cK0lwoA8I6deoEd3d3tUkolg7rcl9qR3gNS09q55o1a4YjR47w83cFMtJVatattWzZ0tKUze+OyxjsbPRFIV8Sf/31V75/Xchj6btDpdewYUP1R2h9LVNSUtQIJ15L03B+CXXSfPj333+r62VNPocyasz6+sl0KPI/P16/osnfalZWFq9dKVx77bWqKVtqPM2b1KJIXzHzfV7D0rt06RKOHj2qQgs/fyWTLicFp3Y6dOiQqvEU/O6wYjWQgipg4cKFavTNl19+qe3fv1978MEHtRo1amhxcXH2LppDjqjbtWuX2uQjOHPmTHX/5MmT6vk33nhDXbslS5Zo//33nzZ06FCtYcOGWkZGhlbdjR8/XgsKCtLWrFmjxcbGWrb09HTLMePGjdPq1aun/f3339r27du1nj17qo007bnnnlMjiI8fP64+W/JYp9NpK1euVM/z2pWd9ahYwWtYvCeffFL97crnb8OGDdqAAQO00NBQNbpd8NoVb+vWrZq7u7v22muvaYcPH9a++eYbzdfXV/vf//5nOYbfHSYMdjb04Ycfqj9KT09PNf3J5s2b7V0kh7R69WoV6Apuo0ePtgxbnzJlihYeHq7C8rXXXqtFR0fbu9gOoajrJtsXX3xhOUb+J/bwww+raTzkf3y33nqrCn+kaffdd59Wv3599Tdaq1Yt9dkyhzrBa1fxYMdrWLzhw4drkZGR6vNXu3Zt9fjIkSOW53ntSvbrr79qbdq0Ud8LLVq00ObOnZvveX53mOjkP9Y1eERERETknNjHjoiIiMhFMNgRERERuQgGOyIiIiIXwWBHRERE5CIY7IiIiIhcBIMdERERkYtgsCMiIiJyEQx2RERERC6CwY6oGurXrx8ef/xxy+MGDRpg1qxZcGQnTpyATqdTa5JWpnvvvRe33HJLhc8jZV28eDEqy5o1a9R7FFw0viBZO1MWSzcYDHBUtvj8Pffcc3j00UdtViYiZ8VgR0TYtm0bHnzwQTiyunXrIjY2Fm3atKnU93n//ffx5ZdfwtH16tVLXY+goKASj3vmmWfw4osvws3NTT2W14wcORLNmjWDXq/PF/AdjSzs/ueff6oQO3ToUERGRsLPzw8dOnTAN998k+/Yp556CgsWLMCxY8fsVl4iR8BgR0SoVasWfH194cgkmERERMDd3b1S30eCUo0aNeDoPD091fWQWrvi/PPPPzh69Chuv/12y76srCz1+5aw1759eziq//77DxcuXEDfvn2xceNGtGvXDj/99JPaP2bMGIwaNQq//fab5fjQ0FAMGjQIn3zyiV3LTWRvDHZELi4tLU19Cfr7+6saj3ffffeKTWESFj799FPcdNNNKvBJU96mTZtw5MgR1YwrtSZSYyShwdqSJUvQqVMneHt7o1GjRnj55ZeRm5ub77yfffYZbr31VnXepk2bYunSpZbn5Yv8rrvuUsHDx8dHPf/FF18U2xS7du1adOvWDV5eXupnk+Y46/eTsj722GOq1iokJEQFoZdeeqlMTbGlOcfhw4dx9dVXq5+7VatWWLVqVaHznj59GnfeeacKjXIeqYGSn0kcPHhQXY9vv/3Wcvz333+vrsH+/fvL3RS7cOFCXHfddapc1r9rqZWUz8SVavsqQ0JCAoYMGaJ+NqmRK1jzZv1Zuv766+Hh4YHJkydj+vTp6jPXuHFjTJw4UT33888/53uNnFd+ZqLqjMGOyMU9/fTTKgDJF+XKlStVINi5c+cVXydfpPLlL0GqRYsWqvnuoYcewvPPP4/t27dD0zQ88sgjluPXr1+vjpcvXQkjEgylSfO1117Ld14JexJwpOblhhtuUEEuKSlJPTdlyhT12hUrVuDAgQOq9kVqYooSExOjXt+1a1f8+++/6tjPP/8cr776ar7jpHlOguiWLVvw1ltv4ZVXXikyeJWkpHMYjUbcdtttqgZNnp8zZw6effbZfK/PyclRtUkBAQHqOm3YsEEFbQkn2dnZ6vq+8847ePjhh3Hq1CmcOXMG48aNw5tvvqmCYnnJe3Xp0gWORIKzhNzVq1fjxx9/xMcff6zCXkES+CX8Fic5OVkFZGsS8uXamQMzUbWkEZHLSk1N1Tw9PbXvv//esu/8+fOaj4+PNnHiRMu++vXra++9957lsfyv4cUXX7Q83rRpk9r3+eefW/Z99913mre3t+Xxtddeq73++uv53v/rr7/WIiMjiz3vpUuX1L4VK1aox0OGDNHGjBlT5M9y/PhxdeyuXbvU48mTJ2vNmzfXjEaj5ZjZs2dr/v7+msFgUI/79u2rXXXVVfnO07VrV+3ZZ58t9pqNHj1aGzp0qOXxlc7xxx9/aO7u7lpMTIzlefl5pKy//PKL5ToULGtWVpb6PcjrzW688UatT58+6loOHDgw3/EFrV69Wr3HhQsXij0mKChI++qrr4p9Xn42689BZYuOjlZl3rp1q2XfgQMH1D7rz9+ZM2fU57a4n23RokXq+b179+bbn5ycrM61Zs2aSvwpiBxb5XZWISK7kqZSqRHq3r27ZZ/UcjRv3vyKr5U+TWbh4eHqtm3btvn2ZWZmIiUlBYGBgarWTGqirGvoZCSmHJOenm7pw2d9XqkFk9eaa2zGjx+v+oNJjeLAgQNVk6g0vxVFavR69uyZr49Z7969cenSJVVrU69evULvJ6TJtqgaotJei4LnkHLIwI6oqCjL81Iua3JtpBlbauysybWxbs6eP3++ZVDDvn37Suw/VxoZGRn5mmHtTa6V9JHs3LmzZZ/UVhbs0yi1dVdddVWRfR2lpk/62M2bNw+tW7fO95w07wr5vBFVVwx2RFQk6dtkZg4YRe2TpkghgUqaWaVZsiDrcGF9DvN5zOcYPHgwTp48ieXLl6umzmuvvRYTJkxQzZS2+DkKvl9VnUOujYSZovqTSX9C6wAofSIl2MnoVQmQFSHN2NJv0dlIsLv55psL7ZcuBdKP7r333lPN/gWZm/StrylRdcM+dkQuTDqaSyiRvl9m8kV/6NAhm7+XDJqIjo5GkyZNCm0SVEpLvpRHjx6N//3vf2pAx9y5c4s8zjygw9TCayI1hlIrVqdOHVQVKYf0GZMgZrZ58+ZC10YGWISFhRW6NuYBDBJKpP/ZCy+8oG6l76HUuFVEx44dix18YQ9SOyeDW3bs2GHZJ58Z6wEgEoKlVq5g/zrpG3rjjTeqfofFTc2zd+9e9XkvWJNHVJ0w2BG5MOmgf//996sBFH///bf64pPQUJagVVpTp07FV199pWrtpBlRmt1khKJMq1GWc8ggD2m2lHPIdBYSnIoiAw0kUMmktDKqVF43bdo0TJo0qVJ+vuIMGDBANZ9KGJUaNxmwIOHMmoQ0qT2TsCLPHz9+XAUVGW0rzcZCBktIk65cr5kzZ6pmbJmbrSJkwIZMeVKQDIiRTULUuXPn1P0rBUCpPZWBM2Zbt25VQU1uzeR5Oa440gVABozIIBz5x4YEvAceeMDShCp+//13dT1l9K6ZBD0JdXK9pKk+Li5ObeYaOjO5tn369Ml3PqLqhsGOyMW9/fbb6stOmrAkhEjfJes+TrYiIUKCmIy8lZGqPXr0UE1m9evXL/U5ZGSphAPp0ybTh8jcdcVNX1G7dm3VZCvBQuZjk2AkIbYsQdIWJET+8ssvqnZNRmVKUCk4Elj6F65bt071+5OmagmrUlbpYyd9DCUQy8/y9ddfqz5o0vdQaiylH5mMEC4vCZQSkKVWrGBNnmwSrGSKFbkvI4xLIn0BrWslpR+bnNe6P5s8X3AKnIJk+hrpjyjz08m1kNo3qck0k4BesBlWRiXL+8yYMUM1T5u3gs3+8lkZO3bsFa4KkWvTyQgKexeCiIgqh9TWygAXmX7G0UkzrQzKkTArIbks5DVPPvmkmkansiexJnJkrLEjInJh0iwstaZlHTBiD9K0+sQTT6ga37KSQSdSG8hQR9Uda+yIiIiIXARr7IiIiIhcBIMdERERkYtgsCMiIiJyEQx2RERERC6CwY6IiIjIRTDYEREREbkIBjsiIiIiF8FgR0REROQiGOyIiIiIXASDHRERERFcw/8Ds7Bm5oesHD0AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "\n",
    "d = 128\n",
    "i = torch.arange(1, d // 2 + 1)\n",
    "\n",
    "base_llama2 = 10_000\n",
    "base_llama3 = 500_000\n",
    "\n",
    "theta_llama2 = base_llama2 ** (-2 * (i - 1) / d)\n",
    "theta_llama3 = base_llama3 ** (-2 * (i - 1) / d)\n",
    "\n",
    "plt.plot(i, theta_llama2, label=\"base = 10,000 (Llama 2)\")\n",
    "plt.plot(i, theta_llama3, label=\"base = 500,000 (Llama 3)\", ls=\"--\")\n",
    "plt.xlabel(\"dimension index i (1 … d/2)\")\n",
    "plt.ylabel(r\"$\\theta_i = \\mathrm{base}^{-2(i-1)/d}$\")\n",
    "plt.legend()\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17f1d883-72a8-4e01-9aeb-a5a136630a1e",
   "metadata": {},
   "source": [
    "- This changed decay rate is to better support longer context lengths without the rotations becoming too much or too strong at higher dimensions (larger positions)\n",
    "- In addition, we introduce a `freq_config` section in the code below that adjusts the frequency; however, we won't be needing it in Llama 3 (only Llama 3.1 and Llama 3.2), so we will revisit this `freq_config` later (it's set to `None` and ignored by default)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6Upl109OOAcu",
   "metadata": {
    "id": "6Upl109OOAcu"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "def precompute_rope_params(head_dim, theta_base=10_000, context_length=4096, freq_config=None):\n",
    "    assert head_dim % 2 == 0, \"Embedding dimension must be even\"\n",
    "\n",
    "    # Compute the inverse frequencies\n",
    "    inv_freq = 1.0 / (theta_base ** (torch.arange(0, head_dim, 2)[: (head_dim // 2)].float() / head_dim))\n",
    "\n",
    "    ################################ NEW ###############################################\n",
    "    # Frequency adjustments\n",
    "    if freq_config is not None:\n",
    "        low_freq_wavelen = freq_config[\"original_context_length\"] / freq_config[\"low_freq_factor\"]\n",
    "        high_freq_wavelen = freq_config[\"original_context_length\"] / freq_config[\"high_freq_factor\"]\n",
    "\n",
    "        wavelen = 2 * torch.pi / inv_freq\n",
    "\n",
    "        inv_freq_llama = torch.where(\n",
    "            wavelen > low_freq_wavelen, inv_freq / freq_config[\"factor\"], inv_freq\n",
    "        )\n",
    "\n",
    "        smooth_factor = (freq_config[\"original_context_length\"] / wavelen - freq_config[\"low_freq_factor\"]) / (\n",
    "            freq_config[\"high_freq_factor\"] - freq_config[\"low_freq_factor\"]\n",
    "        )\n",
    "\n",
    "        smoothed_inv_freq = (\n",
    "            (1 - smooth_factor) * (inv_freq / freq_config[\"factor\"]) + smooth_factor * inv_freq\n",
    "        )\n",
    "\n",
    "        is_medium_freq = (wavelen <= low_freq_wavelen) & (wavelen >= high_freq_wavelen)\n",
    "        inv_freq_llama = torch.where(is_medium_freq, smoothed_inv_freq, inv_freq_llama)\n",
    "        inv_freq = inv_freq_llama\n",
    "    ####################################################################################\n",
    "\n",
    "\n",
    "    # Generate position indices\n",
    "    positions = torch.arange(context_length)\n",
    "\n",
    "    # Compute the angles\n",
    "    angles = positions.unsqueeze(1) * inv_freq.unsqueeze(0)  # Shape: (context_length, head_dim // 2)\n",
    "\n",
    "    # Expand angles to match the head_dim\n",
    "    angles = torch.cat([angles, angles], dim=1)  # Shape: (context_length, head_dim)\n",
    "\n",
    "    # Precompute sine and cosine\n",
    "    cos = torch.cos(angles)\n",
    "    sin = torch.sin(angles)\n",
    "\n",
    "    return cos, sin"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "jJBvO0YMJBXR",
   "metadata": {
    "id": "jJBvO0YMJBXR"
   },
   "source": [
    "- To summarize, what's new so far for Llama 3 compared to Llama 2 are the context length and theta base parameter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "56c37216-e022-4603-be16-f9d3eaeaf4a1",
   "metadata": {
    "id": "56c37216-e022-4603-be16-f9d3eaeaf4a1"
   },
   "outputs": [],
   "source": [
    "# Instantiate RoPE parameters\n",
    "\n",
    "llama_2_context_len = 4096\n",
    "llama_3_context_len = 8192\n",
    "\n",
    "llama_2_theta_base = 10_000\n",
    "llama_3_theta_base = 500_000"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "_V8v6i7MJItU",
   "metadata": {
    "id": "_V8v6i7MJItU"
   },
   "source": [
    "- The usage remains the same as before in Llama 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "dae70c8a-eb18-40f9-a2e5-a6af2a57628b",
   "metadata": {
    "id": "dae70c8a-eb18-40f9-a2e5-a6af2a57628b"
   },
   "outputs": [],
   "source": [
    "# Settings\n",
    "batch_size = 2\n",
    "num_heads = 4\n",
    "head_dim = 16\n",
    "\n",
    "# Instantiate RoPE parameters\n",
    "cos, sin = precompute_rope_params(\n",
    "    head_dim=head_dim,\n",
    "    theta_base=llama_3_theta_base,\n",
    "    context_length=llama_3_context_len\n",
    ")\n",
    "\n",
    "# Dummy query and key tensors\n",
    "torch.manual_seed(123)\n",
    "queries = torch.randn(batch_size, num_heads, llama_3_context_len, head_dim)\n",
    "keys = torch.randn(batch_size, num_heads, llama_3_context_len, head_dim)\n",
    "\n",
    "# Apply rotary position embeddings\n",
    "queries_rot = compute_rope(queries, cos, sin)\n",
    "keys_rot = compute_rope(keys, cos, sin)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd19b75c-cf25-47b8-a010-6733fc0e9a8a",
   "metadata": {
    "id": "cd19b75c-cf25-47b8-a010-6733fc0e9a8a"
   },
   "source": [
    "&nbsp;\n",
    "## 1.3 Grouped-query attention"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "111c7d3f-fded-49e8-a617-9fe67b81dddc",
   "metadata": {
    "id": "111c7d3f-fded-49e8-a617-9fe67b81dddc"
   },
   "source": [
    "- In this section, we replace multi-head attention (MHA) with an alternative mechanism called grouped-query attention (GQA)\n",
    "- In short, one can think of GQA as a more compute- and parameter-efficient version of MHA\n",
    "- In GQA, we reduce the number of key and value projections by sharing them among multiple attention heads\n",
    "- Each attention head still has its unique query, but these queries attend to the same group of keys and values\n",
    "- Below is an illustration of GQA with 2 key-value-groups (kv-groups):\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gpt-to-llama/grouped-query-attention.webp\" width=\"500px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "perAYa2R_KW2",
   "metadata": {
    "id": "perAYa2R_KW2"
   },
   "source": [
    "- The main idea behind GQA is to reduce the number of unique query groups that attend to the key-value pairs, reducing the size of some of the matrix multiplications and the number of parameters in MHA without significantly reducing modeling performance\n",
    "- The GQA code is very similar to MHA (I highlighted the changes below via the \"NEW\" sections)\n",
    "- In short, the main change in GQA is that each query group needs to be repeated to match the number of heads it is associated with, as implemented below"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "842aa71a-4659-424e-8830-392bd6ae86af",
   "metadata": {},
   "source": [
    "- **We also redesign the attention class a bit so it receives the mask through its forward method instead of storing and accessing it as `self.mask`. This lets us build the mask on the fly to reduce memory usage. To foreshadow why: Llama 3.1 can handle sequences of up to 128 k tokens, and precomputing a 128 k × 128 k causal mask would be extremely memory‑intensive, so we avoid it unless absolutely necessary.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9b12e674-ef08-4dd7-8843-615b65b39c91",
   "metadata": {
    "id": "9b12e674-ef08-4dd7-8843-615b65b39c91"
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "\n",
    "class GroupedQueryAttention(nn.Module):\n",
    "    def __init__(\n",
    "            self, d_in, d_out, num_heads,\n",
    "            num_kv_groups,       # NEW\n",
    "            dtype=None\n",
    "        ):\n",
    "        super().__init__()\n",
    "        assert d_out % num_heads == 0, \"d_out must be divisible by num_heads\"\n",
    "        assert num_heads % num_kv_groups == 0, \"num_heads must be divisible by num_kv_groups\"  # NEW\n",
    "\n",
    "        self.d_out = d_out\n",
    "        self.num_heads = num_heads\n",
    "        self.head_dim = d_out // num_heads\n",
    "\n",
    "        ############################# NEW  #############################\n",
    "        # self.W_key = nn.Linear(d_in, d_out, bias=False, dtype=dtype)\n",
    "        # self.W_value = nn.Linear(d_in, d_out, bias=False, dtype=dtype)\n",
    "        self.W_key = nn.Linear(d_in, num_kv_groups * self.head_dim, bias=False, dtype=dtype)\n",
    "        self.W_value = nn.Linear(d_in, num_kv_groups * self.head_dim, bias=False, dtype=dtype)\n",
    "        self.num_kv_groups = num_kv_groups\n",
    "        self.group_size = num_heads // num_kv_groups\n",
    "        ################################################################\n",
    "\n",
    "        self.W_query = nn.Linear(d_in, d_out, bias=False, dtype=dtype)\n",
    "        self.out_proj = nn.Linear(d_out, d_out, bias=False, dtype=dtype)\n",
    "\n",
    "\n",
    "    def forward(self, x, mask=None, cos=None, sin=None):\n",
    "        ##################### NEW  #####################\n",
    "        # The forward method now accepts `mask` instead of accessing it via self.mask.\n",
    "        # Also, we now have cos and sin as input for RoPE\n",
    "        ################################################    \n",
    "        b, num_tokens, d_in = x.shape\n",
    "\n",
    "        queries = self.W_query(x)  # Shape: (b, num_tokens, d_out)\n",
    "        keys = self.W_key(x)  # Shape: (b, num_tokens, num_kv_groups * head_dim)\n",
    "        values = self.W_value(x)  # Shape: (b, num_tokens, num_kv_groups * head_dim)\n",
    "\n",
    "        # Reshape queries, keys, and values\n",
    "        queries = queries.view(b, num_tokens, self.num_heads, self.head_dim)\n",
    "\n",
    "        ##################### NEW  #####################\n",
    "        # keys = keys.view(b, num_tokens, self.num_heads, self.head_dim)\n",
    "        # values = values.view(b, num_tokens, self.num_heads, self.head_dim)\n",
    "        keys = keys.view(b, num_tokens, self.num_kv_groups, self.head_dim)\n",
    "        values = values.view(b, num_tokens, self.num_kv_groups, self.head_dim)\n",
    "        ################################################\n",
    "\n",
    "        # Transpose keys, values, and queries\n",
    "        keys = keys.transpose(1, 2)  # Shape: (b, num_kv_groups, num_tokens, head_dim)\n",
    "        values = values.transpose(1, 2)  # Shape: (b, num_kv_groups, num_tokens, head_dim)\n",
    "        queries = queries.transpose(1, 2)  # Shape: (b, num_heads, num_tokens, head_dim)\n",
    "\n",
    "        ##################### NEW #####################\n",
    "        # Apply RoPE\n",
    "        if cos is not None:\n",
    "            keys = compute_rope(keys, cos, sin)\n",
    "            queries = compute_rope(queries, cos, sin)\n",
    "        ################################################\n",
    "\n",
    "        ##################### NEW  #####################\n",
    "        # Expand keys and values to match the number of heads\n",
    "        # Shape: (b, num_heads, num_tokens, head_dim)\n",
    "\n",
    "        keys = keys.repeat_interleave(self.group_size, dim=1)  # Shape: (b, num_heads, num_tokens, head_dim)\n",
    "        values = values.repeat_interleave(self.group_size, dim=1)  # Shape: (b, num_heads, num_tokens, head_dim)\n",
    "        # For example, before repeat_interleave along dim=1 (query groups):\n",
    "        #   [K1, K2]\n",
    "        # After repeat_interleave (each query group is repeated group_size times):\n",
    "        #   [K1, K1, K2, K2]\n",
    "        # If we used regular repeat instead of repeat_interleave, we'd get:\n",
    "        #   [K1, K2, K1, K2]\n",
    "        ################################################\n",
    "\n",
    "        # Compute scaled dot-product attention (aka self-attention) with a causal mask\n",
    "        # Shape: (b, num_heads, num_tokens, num_tokens)\n",
    "        attn_scores = queries @ keys.transpose(2, 3)  # Dot product for each head\n",
    "\n",
    "        ##################### NEW #####################\n",
    "        # Create mask on the fly\n",
    "        if mask is None:\n",
    "            mask = torch.triu(torch.ones(num_tokens, num_tokens, device=x.device, dtype=torch.bool), diagonal=1)\n",
    "        ################################################\n",
    "    \n",
    "        # Use the mask to fill attention scores\n",
    "        attn_scores.masked_fill_(mask, -torch.inf)\n",
    "\n",
    "        attn_weights = torch.softmax(attn_scores / keys.shape[-1]**0.5, dim=-1)\n",
    "        assert keys.shape[-1] == self.head_dim\n",
    "\n",
    "        # Shape: (b, num_tokens, num_heads, head_dim)\n",
    "        context_vec = (attn_weights @ values).transpose(1, 2)\n",
    "\n",
    "        # Combine heads, where self.d_out = self.num_heads * self.head_dim\n",
    "        context_vec = context_vec.reshape(b, num_tokens, self.d_out)\n",
    "        context_vec = self.out_proj(context_vec)  # optional projection\n",
    "\n",
    "        return context_vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "roAXSwJs9hR8",
   "metadata": {
    "id": "roAXSwJs9hR8"
   },
   "source": [
    "- To illustrate the parameter savings in GQA over MHA, consider the following multi-head attention example from the GPT and Llama 2 code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b4b8f085-349e-4674-a3f0-78fde0664fac",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "b4b8f085-349e-4674-a3f0-78fde0664fac",
    "outputId": "9da09d72-43b1-45af-d46f-6928ea4af33a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W_key: torch.Size([4096, 4096])\n",
      "W_value: torch.Size([4096, 4096])\n",
      "W_query: torch.Size([4096, 4096])\n"
     ]
    }
   ],
   "source": [
    "# Settings\n",
    "batch_size = 1\n",
    "context_len = 3000\n",
    "max_context_len = 8192\n",
    "embed_dim = 4096\n",
    "num_heads = 32\n",
    "\n",
    "\n",
    "example_batch = torch.randn((batch_size, context_len, embed_dim))\n",
    "\n",
    "mha = MultiHeadAttention(\n",
    "    d_in=embed_dim,\n",
    "    d_out=embed_dim,\n",
    "    context_length=max_context_len,\n",
    "    num_heads=num_heads\n",
    ")\n",
    "\n",
    "mha(example_batch)\n",
    "\n",
    "print(\"W_key:\", mha.W_key.weight.shape)\n",
    "print(\"W_value:\", mha.W_value.weight.shape)\n",
    "print(\"W_query:\", mha.W_query.weight.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "IMQtFkcQ9sXC",
   "metadata": {
    "id": "IMQtFkcQ9sXC"
   },
   "source": [
    "- Now, if we use grouped-query attention instead, with 8 kv-groups (that's how many Llama 3 8B uses), we can see that the number of rows of the key and value matrices are reduced by a factor of 4 (because 32 attention heads divided by 8 kv-groups is 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "15e65d3c-7b42-4ed3-bfee-bb09578657bb",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "15e65d3c-7b42-4ed3-bfee-bb09578657bb",
    "outputId": "69709a78-2aaa-4597-8142-2f44eb59753f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W_key: torch.Size([1024, 4096])\n",
      "W_value: torch.Size([1024, 4096])\n",
      "W_query: torch.Size([4096, 4096])\n"
     ]
    }
   ],
   "source": [
    "gqa = GroupedQueryAttention(\n",
    "    d_in=embed_dim,\n",
    "    d_out=embed_dim,\n",
    "    num_heads=num_heads,\n",
    "    num_kv_groups=8,\n",
    ")\n",
    "\n",
    "gqa(example_batch)\n",
    "\n",
    "print(\"W_key:\", gqa.W_key.weight.shape)\n",
    "print(\"W_value:\", gqa.W_value.weight.shape)\n",
    "print(\"W_query:\", gqa.W_query.weight.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a5d4c88-c66a-483b-b4e2-419ff9fd60d5",
   "metadata": {
    "id": "1a5d4c88-c66a-483b-b4e2-419ff9fd60d5"
   },
   "source": [
    "- As a side note, to make the GroupedQueryAttention equivalent to standard multi-head attention, you can set the number of query groups (`num_kv_groups`) equal to the number of heads (`num_heads`)\n",
    "- Lastly, let's compare the number of parameters below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "58f713aa-ac00-4e2f-8247-94609aa01350",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "58f713aa-ac00-4e2f-8247-94609aa01350",
    "outputId": "486dfd9c-9f3a-4b9e-f9a2-35fb43b9a5fb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of parameters:\n",
      "MHA: 67,108,864\n",
      "GQA: 41,943,040\n"
     ]
    }
   ],
   "source": [
    "print(\"Total number of parameters:\")\n",
    "\n",
    "mha_total_params = sum(p.numel() for p in mha.parameters())\n",
    "print(f\"MHA: {mha_total_params:,}\")\n",
    "\n",
    "gqa_total_params = sum(p.numel() for p in gqa.parameters())\n",
    "print(f\"GQA: {gqa_total_params:,}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "78b60dfd-6c0f-41f7-8f0c-8e57116f07f5",
   "metadata": {
    "id": "78b60dfd-6c0f-41f7-8f0c-8e57116f07f5"
   },
   "outputs": [],
   "source": [
    "# Free up memory:\n",
    "del mha\n",
    "del gqa"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fcd8802-2859-45a2-905a-f4fe96629dd9",
   "metadata": {
    "id": "8fcd8802-2859-45a2-905a-f4fe96629dd9"
   },
   "source": [
    "&nbsp;\n",
    "## 1.4 Update the TransformerBlock module"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "KABNccft_YnR",
   "metadata": {
    "id": "KABNccft_YnR"
   },
   "source": [
    "- Next, we update the `TransformerBlock`\n",
    "- Here, we simply swap `MultiHeadAttention` with `GroupedQueryAttention` and add the new RoPE settings\n",
    "- In addition, we also modify the `forward` method so that it receives `mask`, `cos`, and `sin`; since the values for those are the same for each transformer block, we only have to compute them once and then can reuse them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f9fa8eb4-7196-4dee-aec6-0dcbc70921c4",
   "metadata": {
    "id": "f9fa8eb4-7196-4dee-aec6-0dcbc70921c4"
   },
   "outputs": [],
   "source": [
    "class TransformerBlock(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        self.att =  GroupedQueryAttention(  # MultiHeadAttention(\n",
    "            d_in=cfg[\"emb_dim\"],\n",
    "            d_out=cfg[\"emb_dim\"],\n",
    "            num_heads=cfg[\"n_heads\"],\n",
    "            num_kv_groups=cfg[\"n_kv_groups\"],  # NEW\n",
    "            dtype=cfg[\"dtype\"]\n",
    "        )\n",
    "        self.ff = FeedForward(cfg)\n",
    "        self.norm1 = RMSNorm(cfg[\"emb_dim\"], eps=1e-5)\n",
    "        self.norm2 = RMSNorm(cfg[\"emb_dim\"], eps=1e-5)\n",
    "\n",
    "    def forward(self, x, mask=None, cos=None, sin=None):\n",
    "        ##################### NEW  #####################\n",
    "        # The forward method now accepts `mask` instead of accessing it via self.mask.\n",
    "        # Also, we now have cos and sin as input for RoPE\n",
    "        ################################################\n",
    "        # Shortcut connection for attention block\n",
    "        shortcut = x\n",
    "        x = self.norm1(x)\n",
    "        x = self.att(x.to(torch.bfloat16), mask, cos, sin)   # Shape [batch_size, num_tokens, emb_size]\n",
    "        x = x + shortcut  # Add the original input back\n",
    "\n",
    "        # Shortcut connection for feed-forward block\n",
    "        shortcut = x\n",
    "        x = self.norm2(x)\n",
    "        x = self.ff(x.to(torch.bfloat16))\n",
    "        x = x + shortcut  # Add the original input back\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd921ab5-c48c-4c52-bf41-b847b3b822b9",
   "metadata": {
    "id": "fd921ab5-c48c-4c52-bf41-b847b3b822b9"
   },
   "source": [
    "&nbsp;\n",
    "## 1.5 Defining the model class"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "M_tLAq_r_llN",
   "metadata": {
    "id": "M_tLAq_r_llN"
   },
   "source": [
    "- When setting up the model class, we technically don't have to do much; we just update the name to `Llama3Model`\n",
    "- However, since we now pass the `mask`, `cos`, and `sin` to the transformer blocks, we also have to add them here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "475755d6-01f7-4e6e-ad9a-cec6f031ebf6",
   "metadata": {
    "id": "475755d6-01f7-4e6e-ad9a-cec6f031ebf6"
   },
   "outputs": [],
   "source": [
    "# class Llama2Model(nn.Module):\n",
    "class Llama3Model(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        self.tok_emb = nn.Embedding(cfg[\"vocab_size\"], cfg[\"emb_dim\"], dtype=cfg[\"dtype\"])\n",
    "\n",
    "        self.trf_blocks = nn.Sequential(\n",
    "            *[TransformerBlock(cfg) for _ in range(cfg[\"n_layers\"])])\n",
    "\n",
    "        self.final_norm = RMSNorm(cfg[\"emb_dim\"], eps=1e-5)\n",
    "        self.out_head = nn.Linear(cfg[\"emb_dim\"], cfg[\"vocab_size\"], bias=False, dtype=cfg[\"dtype\"])\n",
    "\n",
    "        #################### NEW #####################\n",
    "        cos, sin = precompute_rope_params(\n",
    "            head_dim=cfg[\"emb_dim\"] // cfg[\"n_heads\"],\n",
    "            theta_base=cfg[\"rope_base\"],\n",
    "            context_length=cfg[\"context_length\"],\n",
    "            freq_config=cfg[\"rope_freq\"]\n",
    "        )\n",
    "        \n",
    "        self.register_buffer(\"cos\", cos, persistent=False)\n",
    "        self.register_buffer(\"sin\", sin, persistent=False)\n",
    "        ##############################################\n",
    "\n",
    "        self.cfg = cfg\n",
    "\n",
    "    def forward(self, in_idx):\n",
    "        tok_embeds = self.tok_emb(in_idx)\n",
    "        x = tok_embeds\n",
    "\n",
    "        #################### NEW #####################\n",
    "        num_tokens = x.shape[1]\n",
    "        mask = torch.triu(torch.ones(num_tokens, num_tokens, device=x.device, dtype=torch.bool), diagonal=1)\n",
    "        ##############################################\n",
    "        \n",
    "        for block in self.trf_blocks:\n",
    "            x = block(x, mask, self.cos, self.sin)\n",
    "        x = self.final_norm(x)\n",
    "        logits = self.out_head(x.to(self.cfg[\"dtype\"]))\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bc94940-aaeb-45b9-9399-3a69b8043e60",
   "metadata": {
    "id": "4bc94940-aaeb-45b9-9399-3a69b8043e60"
   },
   "source": [
    "&nbsp;\n",
    "## 2. Initialize model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "HoGGRAGykQTE",
   "metadata": {
    "id": "HoGGRAGykQTE"
   },
   "source": [
    "- Now we can define a Llama 3 config file (the Llama 2 config file is shown for comparison)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e0564727-2d35-4f0c-b0fc-cde1e9134a18",
   "metadata": {
    "id": "e0564727-2d35-4f0c-b0fc-cde1e9134a18"
   },
   "outputs": [],
   "source": [
    "LLAMA2_CONFIG_7B = {\n",
    "    \"vocab_size\": 32_000,    # Vocabulary size\n",
    "    \"context_length\": 4096,  # Context length\n",
    "    \"emb_dim\": 4096,         # Embedding dimension\n",
    "    \"n_heads\": 32,           # Number of attention heads\n",
    "    \"n_layers\": 32,          # Number of layers\n",
    "    \"hidden_dim\": 11_008,    # Size of the intermediate dimension in FeedForward\n",
    "    \"dtype\": torch.bfloat16  # Lower-precision dtype to reduce memory usage\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2ad90f82-15c7-4806-b509-e45b56f57db5",
   "metadata": {
    "id": "2ad90f82-15c7-4806-b509-e45b56f57db5"
   },
   "outputs": [],
   "source": [
    "LLAMA3_CONFIG_8B = {\n",
    "    \"vocab_size\": 128_256,   # NEW: Larger vocabulary size\n",
    "    \"context_length\": 8192,  # NEW: Larger context length\n",
    "    \"emb_dim\": 4096,         # Embedding dimension\n",
    "    \"n_heads\": 32,           # Number of attention heads\n",
    "    \"n_layers\": 32,          # Number of layers\n",
    "    \"hidden_dim\": 14_336,    # NEW: Larger size of the intermediate dimension in FeedForward\n",
    "    \"n_kv_groups\": 8,        # NEW: Key-Value groups for grouped-query attention\n",
    "    \"rope_base\": 500_000.0,  # NEW: The base in RoPE's \"theta\" was increased to 500_000\n",
    "    \"rope_freq\": None,       # NEW: Additional configuration for adjusting the RoPE frequencies\n",
    "    \"dtype\": torch.bfloat16  # Lower-precision dtype to reduce memory usage\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "FAP7fiBzkaBz",
   "metadata": {
    "id": "FAP7fiBzkaBz"
   },
   "source": [
    "- Using these settings, we can now initialize a Llama 3 8B model\n",
    "- Note that this requires ~34 GB of memory (for comparison, Llama 2 7B required ~26 GB of memory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "7004d785-ac9a-4df5-8760-6807fc604686",
   "metadata": {
    "id": "7004d785-ac9a-4df5-8760-6807fc604686"
   },
   "outputs": [],
   "source": [
    "model = Llama3Model(LLAMA3_CONFIG_8B)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8056a521-91a6-440f-8473-591409c3177b",
   "metadata": {},
   "source": [
    "- Let's now compute the number of trainable parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6079f747-8f20-4c6b-8d38-7156f1101729",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6079f747-8f20-4c6b-8d38-7156f1101729",
    "outputId": "0a8cd23b-d9fa-4c2d-ca63-3fc79bc4de0d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of parameters: 8,030,261,248\n"
     ]
    }
   ],
   "source": [
    "total_params = sum(p.numel() for p in model.parameters())\n",
    "print(f\"Total number of parameters: {total_params:,}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "Bx14NtzWk2wj",
   "metadata": {
    "id": "Bx14NtzWk2wj"
   },
   "source": [
    "- As shown above, the model contains 8 billion parameters\n",
    "- Additionally, we can calculate the memory requirements for this model using the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "0df1c79e-27a7-4b0f-ba4e-167fe107125a",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0df1c79e-27a7-4b0f-ba4e-167fe107125a",
    "outputId": "3425e9ce-d8c0-4b37-bded-a2c60b66a41a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "float32 (PyTorch default): 59.84 GB\n",
      "bfloat16: 29.92 GB\n"
     ]
    }
   ],
   "source": [
    "def model_memory_size(model, input_dtype=torch.float32):\n",
    "    total_params = 0\n",
    "    total_grads = 0\n",
    "    for param in model.parameters():\n",
    "        # Calculate total number of elements per parameter\n",
    "        param_size = param.numel()\n",
    "        total_params += param_size\n",
    "        # Check if gradients are stored for this parameter\n",
    "        if param.requires_grad:\n",
    "            total_grads += param_size\n",
    "\n",
    "    # Calculate buffer size (non-parameters that require memory)\n",
    "    total_buffers = sum(buf.numel() for buf in model.buffers())\n",
    "\n",
    "    # Size in bytes = (Number of elements) * (Size of each element in bytes)\n",
    "    # We assume parameters and gradients are stored in the same type as input dtype\n",
    "    element_size = torch.tensor(0, dtype=input_dtype).element_size()\n",
    "    total_memory_bytes = (total_params + total_grads + total_buffers) * element_size\n",
    "\n",
    "    # Convert bytes to gigabytes\n",
    "    total_memory_gb = total_memory_bytes / (1024**3)\n",
    "\n",
    "    return total_memory_gb\n",
    "\n",
    "print(f\"float32 (PyTorch default): {model_memory_size(model, input_dtype=torch.float32):.2f} GB\")\n",
    "print(f\"bfloat16: {model_memory_size(model, input_dtype=torch.bfloat16):.2f} GB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "zudd-5PulKFL",
   "metadata": {
    "id": "zudd-5PulKFL"
   },
   "source": [
    "- Lastly, we can also transfer the model to an NVIDIA or Apple Silicon GPU if applicable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "a4c50e19-1402-45b6-8ccd-9077b2ba836d",
   "metadata": {
    "id": "a4c50e19-1402-45b6-8ccd-9077b2ba836d"
   },
   "outputs": [],
   "source": [
    "if torch.cuda.is_available():\n",
    "    device = torch.device(\"cuda\")\n",
    "elif torch.backends.mps.is_available():\n",
    "    device = torch.device(\"mps\")\n",
    "else:\n",
    "    device = torch.device(\"cpu\")\n",
    "\n",
    "model.to(device);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dc64a06-27dc-46ec-9e6d-1700a8227d34",
   "metadata": {
    "id": "5dc64a06-27dc-46ec-9e6d-1700a8227d34"
   },
   "source": [
    "&nbsp;\n",
    "## 3. Load tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0eb30f0c-6144-4bed-87d9-6b2bac377005",
   "metadata": {
    "id": "0eb30f0c-6144-4bed-87d9-6b2bac377005"
   },
   "source": [
    "- In this section, we are going to load the tokenizer for the model\n",
    "- Llama 2 used Google's [SentencePiece](https://github.com/google/sentencepiece) tokenizer instead of OpenAI's BPE tokenizer based on the [Tiktoken](https://github.com/openai/tiktoken) library\n",
    "- Llama 3, however, reverted back to using the BPE tokenizer from Tiktoken; specifically, it uses the GPT-4 tokenizer with an extended vocabulary\n",
    "- You can find the original Tiktoken-adaptation by Meta AI [here](https://github.com/meta-llama/llama3/blob/main/llama/tokenizer.py) in their official Llama 3 repository\n",
    "- Below, I rewrote the tokenizer code to make it more readable and minimal for this notebook (but the behavior should be similar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "5f390cbf-8f92-46dc-afe3-d90b5affae10",
   "metadata": {
    "id": "5f390cbf-8f92-46dc-afe3-d90b5affae10"
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "import tiktoken\n",
    "from tiktoken.load import load_tiktoken_bpe\n",
    "\n",
    "\n",
    "class Tokenizer:\n",
    "    \"\"\"Thin wrapper around tiktoken that keeps track of Llama-3 special IDs.\"\"\"\n",
    "    def __init__(self, model_path):\n",
    "        if not os.path.isfile(model_path):\n",
    "            raise FileNotFoundError(model_path)\n",
    "\n",
    "        mergeable = load_tiktoken_bpe(model_path)\n",
    "\n",
    "        # hard-coded from Meta's tokenizer.json\n",
    "        self.special = {\n",
    "            \"<|begin_of_text|>\": 128000,\n",
    "            \"<|end_of_text|>\": 128001,\n",
    "            \"<|start_header_id|>\": 128006,\n",
    "            \"<|end_header_id|>\": 128007,\n",
    "            \"<|eot_id|>\": 128009,\n",
    "        }\n",
    "        self.special.update({f\"<|reserved_{i}|>\": 128002 + i\n",
    "                             for i in range(256)\n",
    "                             if 128002 + i not in self.special.values()})\n",
    "\n",
    "        self.model = tiktoken.Encoding(\n",
    "            name=Path(model_path).name,\n",
    "            pat_str=r\"(?i:'s|'t|'re|'ve|'m|'ll|'d)\"\n",
    "                    r\"|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+\"\n",
    "                    r\"|\\p{N}{1,3}\"\n",
    "                    r\"| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*\"\n",
    "                    r\"|\\s*[\\r\\n]+\"\n",
    "                    r\"|\\s+(?!\\S)\"\n",
    "                    r\"|\\s+\",\n",
    "            mergeable_ranks=mergeable,\n",
    "            special_tokens=self.special,\n",
    "        )\n",
    "\n",
    "    def encode(self, text, bos=False, eos=False):\n",
    "        ids = ([self.special[\"<|begin_of_text|>\"]] if bos else []) \\\n",
    "              + self.model.encode(text)\n",
    "        if eos:\n",
    "            ids.append(self.special[\"<|end_of_text|>\"])\n",
    "        return ids\n",
    "\n",
    "    def decode(self, ids):\n",
    "        return self.model.decode(ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a1509f8-8778-4fec-ba32-14d95c646167",
   "metadata": {
    "id": "0a1509f8-8778-4fec-ba32-14d95c646167"
   },
   "source": [
    "- Meta AI shared the original Llama 3 model weights and tokenizer vocabulary on the Hugging Face Hub\n",
    "- We will first download the tokenizer vocabulary from the Hub and load it into the code above"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "KbnlzsbYmJU6",
   "metadata": {
    "id": "KbnlzsbYmJU6"
   },
   "source": [
    "- Please note that Meta AI requires that you accept the Llama 3 licensing terms before you can download the files; to do this, you have to create a Hugging Face Hub account and visit the [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) repository to accept the terms\n",
    "- Next, you will need to create an access token; to generate an access token with READ permissions, click on the profile picture in the upper right and click on \"Settings\"\n",
    "\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gpt-to-llama/settings.webp?1\" width=\"300px\">\n",
    "\n",
    "- Then, create and copy the access token so you can copy & paste it into the next code cell\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gpt-to-llama/access-token.webp?1\" width=\"600px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "3357a230-b678-4691-a238-257ee4e80185",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3357a230-b678-4691-a238-257ee4e80185",
    "outputId": "a3652def-ea7f-46fb-f293-2a59affb71a0"
   },
   "outputs": [],
   "source": [
    "from huggingface_hub import login\n",
    "import json\n",
    "\n",
    "with open(\"config.json\", \"r\") as config_file:\n",
    "    config = json.load(config_file)\n",
    "    access_token = config[\"HF_ACCESS_TOKEN\"]\n",
    "\n",
    "login(token=access_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "IxGh6ZYQo0VN",
   "metadata": {
    "id": "IxGh6ZYQo0VN"
   },
   "source": [
    "- After login via the access token, which is necessary to verify that we accepted the Llama 3 licensing terms, we can now download the tokenizer vocabulary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "69714ea8-b9b8-4687-8392-f3abb8f93a32",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "69714ea8-b9b8-4687-8392-f3abb8f93a32",
    "outputId": "c9836ba8-5176-4dd5-b618-6cc36fdbe1f0"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "685326b4fd014ff689e928f4200f5182",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "original/tokenizer.model:   0%|          | 0.00/2.18M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from huggingface_hub import hf_hub_download\n",
    "\n",
    "tokenizer_file_path = hf_hub_download(\n",
    "    repo_id=\"meta-llama/Meta-Llama-3-8B\",\n",
    "    filename=\"original/tokenizer.model\",\n",
    "    local_dir=\"Llama-3-8B\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "F8BH1Nk0AYCS",
   "metadata": {
    "id": "F8BH1Nk0AYCS"
   },
   "source": [
    "- Note that for using Llama 3 files, we may need the `blobfile` package, which is used when handling datasets or models stored in cloud storage solutions like Google Cloud Storage (GCS), Azure Blob Storage, or Amazon S3\n",
    "- You can install this dependency by uncommenting and executing the `pip` command below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "5dm6Oz7uAytV",
   "metadata": {
    "id": "5dm6Oz7uAytV"
   },
   "outputs": [],
   "source": [
    "# pip install blobfile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "8b8c0ce6-a6fb-4b8a-8de2-ee7bb7646fd0",
   "metadata": {
    "id": "8b8c0ce6-a6fb-4b8a-8de2-ee7bb7646fd0"
   },
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer(tokenizer_file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "NVhmFeX3pT_M",
   "metadata": {
    "id": "NVhmFeX3pT_M"
   },
   "source": [
    "- We can now use the `generate` function to have the Llama 3 model generate new text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e0a2b5cd-6cba-4d72-b8ff-04d8315d483e",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "e0a2b5cd-6cba-4d72-b8ff-04d8315d483e",
    "outputId": "990d7b74-cb35-476b-d8bd-d544006e00f4",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output text:\n",
      " Every effort_dead aeros Ingredients başında.extensionégor clangmissions güc như submodule.and report官方%，.Reader(\",\");\n",
      "ामल ندار Parliamentary !!! HigginsDynamicZhamincus_beam cyc......\n",
      "\n",
      " haciendo\n"
     ]
    }
   ],
   "source": [
    "from previous_chapters import generate, text_to_token_ids, token_ids_to_text\n",
    "# If the `previous_chapters.py` file is not available locally,\n",
    "# you can import it from the `llms-from-scratch` PyPI package.\n",
    "# For details, see: https://github.com/rasbt/LLMs-from-scratch/tree/main/pkg\n",
    "# E.g.,\n",
    "# from llms_from_scratch.ch05 import generate, text_to_token_ids, token_ids_to_text\n",
    "\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "token_ids = generate(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(\"Every effort\", tokenizer).to(device),\n",
    "    max_new_tokens=30,\n",
    "    context_size=LLAMA3_CONFIG_8B[\"context_length\"],\n",
    "    top_k=1,\n",
    "    temperature=0.\n",
    ")\n",
    "\n",
    "print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93WTtAA5paYV",
   "metadata": {
    "id": "93WTtAA5paYV"
   },
   "source": [
    "- Of course, as we can see above, the text is nonsensical since we haven't trained the Llama 3 model yet\n",
    "- In the next section, instead of training it ourselves, which would cost tens to hundreds of thousands of dollars, we load the pretrained weights from Meta AI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f63cc248-1d27-4eb6-aa50-173b436652f8",
   "metadata": {
    "id": "f63cc248-1d27-4eb6-aa50-173b436652f8"
   },
   "source": [
    "&nbsp;\n",
    "## 4. Load pretrained weights"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aKeN7rUfqZMI",
   "metadata": {
    "id": "aKeN7rUfqZMI"
   },
   "source": [
    "- We are loading the [\"meta-llama/Meta-Llama-3-8B\"](https://huggingface.co/meta-llama/Meta-Llama-3-8B) base model below, which is a simple text completion model before finetuning\n",
    "- Alternatively, you can load the instruction-finetuned and aligned [\"meta-llama/Meta-Llama-3-8B-Instruct\"](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) model by modifying the string in the next code cell accordingly\n",
    "- Combined, the weight files are about 16 GB large"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "5fa9c06c-7a53-4b4d-9ce4-acc027322ee4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 145,
     "referenced_widgets": [
      "f3788acce34f4956b0727b58d0cf38c6",
      "6022a9426683420690d9b41a0ca4f870",
      "e9aba3d53b4d45c485a7aad649c7b465",
      "f1a12d7929db4309b9881853135359fc",
      "58c9dec75a3346b1b787f88dd510d254",
      "9492edc02dee456f840325d913fa4e4f",
      "66dc94b23556499f985f8accbb1f89cb",
      "7c6658cfff1a4d27af3de148184f77d9",
      "7266a729edfb4a44b5b1c67dc79be146",
      "76dbab4873f342019c5d7624ae2c9775",
      "3cea4b431147441a8d9bd872811d5974",
      "8ae98969541849efa356cf912ac39b1e",
      "f9373112649945e3b446c3e1ec274dc1",
      "d49791082a304ade95c185c79fae1f41",
      "616e383bb3d442bcb6edb2721a8180b6",
      "87f474861e54432e9d533e0a89bb77da",
      "e805bb6dfee34dab8870f4618d8bffdb",
      "be3e9bf271f04eb0b119659e1af3a0ea",
      "00148825ce0248b7a23eb28e3eca6749",
      "f1a9b0c2431640298a6c1b258298b12d",
      "8ba9f009e92a46fcbcbb401dc444f12e",
      "d74186bb74d142dfb683fa347b6990f7",
      "9bb60a5a3710463ebe3a17f8d2a446be",
      "0a08fb81165748748ccb080e6df0600f",
      "603690f543114a7fb6aebd433c80bdc3",
      "773b802daed942f5a11f3eab3b83be08",
      "7989003a613e45f780d3f800e121543a",
      "9d49589118f5432cac49650251046429",
      "f114549fe8ce49638a791ca2fecb2d89",
      "0aa155b794a8426aa265f4a7670f43ad",
      "a06fbde549cc47fdaddfbdb82d35d823",
      "172c0c6955e1428b999dcb2d133704cd",
      "1bf7108774c34016a2193e2cd7639b7d",
      "ed28e180d94a4b7aa548581612e31232",
      "ff4338faded5494da1ccb660e1c441ed",
      "b46a08cf4929422eb0f76d8d9af11249",
      "f049eb4a50f54c34912ca959d2eaf353",
      "80dfd3e80ceb444a83ec1fd65f9af80e",
      "519147a10b984befbd0f255f78c1f66a",
      "562e82438dbe41b793ff488b8447c5bf",
      "1da83719e47c4196b06f3aa32056b560",
      "c4a2c88326d14fbca87cfde073755a2e",
      "f0ab5a46cbb0444c88ed137d8a95002b",
      "f8f28ac0e149428f9fef42373c6a87d0"
     ]
    },
    "id": "5fa9c06c-7a53-4b4d-9ce4-acc027322ee4",
    "outputId": "c05118ce-9f81-41c8-a1f2-72caa932ae86"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3af9f77314b14682bbdd1c4921cd193e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00001-of-00004.safetensors:   0%|          | 0.00/4.98G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7aeb092ad0a14b5e9aaf33bea4751490",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00002-of-00004.safetensors:   0%|          | 0.00/5.00G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "20adbc86984344a39a55f012b8c18d68",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00003-of-00004.safetensors:   0%|          | 0.00/4.92G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e6bb24f8ca4344dfb3870fca8c90e4fb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00004-of-00004.safetensors:   0%|          | 0.00/1.17G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from safetensors.torch import load_file\n",
    "\n",
    "combined_weights = {}\n",
    "\n",
    "for i in range(1, 5):\n",
    "    weights_file = hf_hub_download(\n",
    "        repo_id=\"meta-llama/Meta-Llama-3-8B\",\n",
    "        filename=f\"model-0000{i}-of-00004.safetensors\",\n",
    "        local_dir=\"Llama-3-8B\"\n",
    "    )\n",
    "    current_weights = load_file(weights_file)\n",
    "    combined_weights.update(current_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "-15SJ7btq2zE",
   "metadata": {
    "id": "-15SJ7btq2zE"
   },
   "source": [
    "- The `weights` contains the following tensors (only the first 15 are shown for simplicity):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "ee26bd0b-fea9-4924-97f7-409c14f28e49",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ee26bd0b-fea9-4924-97f7-409c14f28e49",
    "outputId": "2fbc2786-677f-4fea-9472-5fb8542ff14b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['model.embed_tokens.weight',\n",
       " 'model.layers.0.input_layernorm.weight',\n",
       " 'model.layers.0.mlp.down_proj.weight',\n",
       " 'model.layers.0.mlp.gate_proj.weight',\n",
       " 'model.layers.0.mlp.up_proj.weight',\n",
       " 'model.layers.0.post_attention_layernorm.weight',\n",
       " 'model.layers.0.self_attn.k_proj.weight',\n",
       " 'model.layers.0.self_attn.o_proj.weight',\n",
       " 'model.layers.0.self_attn.q_proj.weight',\n",
       " 'model.layers.0.self_attn.v_proj.weight',\n",
       " 'model.layers.1.input_layernorm.weight',\n",
       " 'model.layers.1.mlp.down_proj.weight',\n",
       " 'model.layers.1.mlp.gate_proj.weight',\n",
       " 'model.layers.1.mlp.up_proj.weight',\n",
       " 'model.layers.1.post_attention_layernorm.weight']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(combined_weights.keys())[:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "UeeSpnunrDFB",
   "metadata": {
    "id": "UeeSpnunrDFB"
   },
   "source": [
    "- The following function, modeled after the `load_weights_into_gpt` function in [chapter 5](../01_main-chapter-code/ch05.ipynb), loads the pretrained weights into our Llama 3 model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "3820e2a7-4f26-41bc-953b-f3879b0aff65",
   "metadata": {
    "id": "3820e2a7-4f26-41bc-953b-f3879b0aff65"
   },
   "outputs": [],
   "source": [
    "def assign(left, right, tensor_name=\"unknown\"):\n",
    "    if left.shape != right.shape:\n",
    "        raise ValueError(f\"Shape mismatch in tensor '{tensor_name}'. Left: {left.shape}, Right: {right.shape}\")\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        if isinstance(right, torch.Tensor):\n",
    "            left.copy_(right)\n",
    "        else:\n",
    "            left.copy_(torch.as_tensor(right, dtype=left.dtype, device=left.device))\n",
    "\n",
    "    return left \n",
    "\n",
    "\n",
    "def load_weights_into_llama(model, param_config, params):\n",
    "    model.tok_emb.weight = assign(model.tok_emb.weight, params[\"model.embed_tokens.weight\"], \"model.embed_tokens.weight\")\n",
    "\n",
    "    for l in range(param_config[\"n_layers\"]):\n",
    "\n",
    "        # Load attention weights\n",
    "        model.trf_blocks[l].att.W_query.weight = assign(\n",
    "            model.trf_blocks[l].att.W_query.weight,\n",
    "            params[f\"model.layers.{l}.self_attn.q_proj.weight\"],\n",
    "            f\"model.layers.{l}.self_attn.q_proj.weight\"\n",
    "        )\n",
    "        model.trf_blocks[l].att.W_key.weight = assign(\n",
    "            model.trf_blocks[l].att.W_key.weight,\n",
    "            params[f\"model.layers.{l}.self_attn.k_proj.weight\"],\n",
    "            f\"model.layers.{l}.self_attn.k_proj.weight\"\n",
    "        )\n",
    "        model.trf_blocks[l].att.W_value.weight = assign(\n",
    "            model.trf_blocks[l].att.W_value.weight,\n",
    "            params[f\"model.layers.{l}.self_attn.v_proj.weight\"],\n",
    "            f\"model.layers.{l}.self_attn.v_proj.weight\"\n",
    "        )\n",
    "        model.trf_blocks[l].att.out_proj.weight = assign(\n",
    "            model.trf_blocks[l].att.out_proj.weight,\n",
    "            params[f\"model.layers.{l}.self_attn.o_proj.weight\"],\n",
    "            f\"model.layers.{l}.self_attn.o_proj.weight\"\n",
    "        )\n",
    "        model.trf_blocks[l].norm1.weight = assign(\n",
    "            model.trf_blocks[l].norm1.weight,\n",
    "            params[f\"model.layers.{l}.input_layernorm.weight\"],\n",
    "            f\"model.layers.{l}.input_layernorm.weight\"\n",
    "        )\n",
    "\n",
    "        # Load FeedForward weights\n",
    "        model.trf_blocks[l].ff.fc1.weight = assign(\n",
    "            model.trf_blocks[l].ff.fc1.weight,\n",
    "            params[f\"model.layers.{l}.mlp.gate_proj.weight\"],\n",
    "            f\"model.layers.{l}.mlp.gate_proj.weight\"\n",
    "        )\n",
    "        model.trf_blocks[l].ff.fc2.weight = assign(\n",
    "            model.trf_blocks[l].ff.fc2.weight,\n",
    "            params[f\"model.layers.{l}.mlp.up_proj.weight\"],\n",
    "            f\"model.layers.{l}.mlp.up_proj.weight\"\n",
    "        )\n",
    "        model.trf_blocks[l].ff.fc3.weight = assign(\n",
    "            model.trf_blocks[l].ff.fc3.weight,\n",
    "            params[f\"model.layers.{l}.mlp.down_proj.weight\"],\n",
    "            f\"model.layers.{l}.mlp.down_proj.weight\"\n",
    "        )\n",
    "        model.trf_blocks[l].norm2.weight = assign(\n",
    "            model.trf_blocks[l].norm2.weight,\n",
    "            params[f\"model.layers.{l}.post_attention_layernorm.weight\"],\n",
    "            f\"model.layers.{l}.post_attention_layernorm.weight\"\n",
    "        )\n",
    "\n",
    "    # Load output layer weights\n",
    "    model.final_norm.weight = assign(model.final_norm.weight, params[\"model.norm.weight\"], \"model.norm.weight\")\n",
    "\n",
    "    if \"lm_head.weight\" in params.keys():\n",
    "        model.out_head.weight = assign(model.out_head.weight, params[\"lm_head.weight\"], \"lm_head.weight\")\n",
    "    else:\n",
    "        model.out_head.weight = model.tok_emb.weight\n",
    "        print(\"Model uses weight tying.\")\n",
    "\n",
    "\n",
    "load_weights_into_llama(model, LLAMA3_CONFIG_8B, combined_weights)\n",
    "model.to(device);\n",
    "del combined_weights  # free up memory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "TDuv_Us2rNvk",
   "metadata": {
    "id": "TDuv_Us2rNvk"
   },
   "source": [
    "- Next, we are ready to use the model for text generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "240987e8-a023-462e-9376-9edfb27559ec",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "240987e8-a023-462e-9376-9edfb27559ec",
    "outputId": "6dab0e56-40a8-45db-a096-ab2b9ee97a69"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output text:\n",
      " Every effort has been made to trace copyright holders and to obtain their permission for the use of copyright material. The publisher apologizes for any\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "token_ids = generate(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(\"Every effort\", tokenizer).to(device),\n",
    "    max_new_tokens=25,\n",
    "    context_size=LLAMA3_CONFIG_8B[\"context_length\"],\n",
    "    top_k=1,\n",
    "    temperature=0.\n",
    ")\n",
    "\n",
    "print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1203041e-4794-4157-a978-3ce80909da44",
   "metadata": {
    "id": "1203041e-4794-4157-a978-3ce80909da44"
   },
   "source": [
    "&nbsp;\n",
    "## 5. Using the instruction-finetuned model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "akyo7WNyF_YL",
   "metadata": {
    "id": "akyo7WNyF_YL"
   },
   "source": [
    "- Above, we used the pretrained base model; if you want to use a model capable of following instructions, use the `\"meta-llama/Llama-3-8B-Instruct\"` model instead, as shown below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "hdA-xjjdS26J",
   "metadata": {
    "id": "hdA-xjjdS26J"
   },
   "outputs": [],
   "source": [
    "# to free up memory\n",
    "\n",
    "import gc\n",
    "\n",
    "del model\n",
    "\n",
    "gc.collect()  # Run Python garbage collector\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "nbvAV7vaz6yc",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 145,
     "referenced_widgets": [
      "409470784b6346a981920350de4f6f28",
      "9ba6a11ffd194bf9a0900f52a7ed4d4f",
      "acae8bbbb4a84ed49be72fecd11fb052",
      "e8a4b441281b4038bb0204d093411f68",
      "bdf8b693821344fc97918e6cbc31c8bf",
      "97e8877869cd4be68ff38ce745be5045",
      "cc3da88e93c4499993b7bbb7d3064326",
      "0d51fdc2c416474da04079db6579890f",
      "c4598300a77b4667b1117f9499f5ccb7",
      "77606cd2fe1b4d33a91ede944bb1dec0",
      "f1ba439c26d64c90af2f162c74348405",
      "d598f094c3ce4daeab19fac8094cba7e",
      "0afc2d23514b45c9890b5d2ee4e6fa0b",
      "3da5d38bf3314d3eaa7cedebae41c076",
      "55e6b727a4594078beb3853cc1891308",
      "f17fa78263414ef8b414c7bf3ac03192",
      "e8b187b40ec14db3af17a380830a35bf",
      "e94ca32eaa9f4714a3b05a5fdf24d02b",
      "3edd464991204b8690eae02f10b4cc00",
      "ac1e34f4bd6c420bb6cc2fdde5f3ed4d",
      "1cd5e07cad35450182004952de32c8e7",
      "a63351a6715643378491ba831b3fb05d",
      "98b4680141ee423bb5e43c47613d8440",
      "b02ffefca3f34252914e76f4a8a467dc",
      "31d27bf34a74432f8e0dbfe9ecb76130",
      "a3137f3669b54e84be91010c9654d985",
      "5a2886564d3f40ceaa30b743dbe81f45",
      "15ea8fcfe097471e8fc9502a162f5904",
      "c779e80c50ba4434bfa1d326c5cc9b0f",
      "eb94612785e64552aea8674dc8647a93",
      "279cffe683fe4e7383062162e07ed9ed",
      "6176990205cc499f8995c71fc6b9d4df",
      "66c23ae98bcc45f18fc5c91e0e73c3e4",
      "05b502e1e3a9436297dafbb1ce7af722",
      "25977b0d89084703ad787fe9208b5aad",
      "71a84ee5fc964ec89ff2832c84735cc2",
      "6aed783eccb942318e6384e253ad4924",
      "84c34bfecda64391a609e19f131d51d4",
      "20ecac7c646b45938ed393cb20977c37",
      "ebe04aeaaac042aaaa0885992e45793d",
      "ca81071ab07446df96795a482ce0c630",
      "e0550cab24c7492787af40dc4b8576bf",
      "7015bf6f85954036aaf8cc4f1c44ea0f",
      "2a2ba3d065634484a932b8d3c212af56"
     ]
    },
    "id": "nbvAV7vaz6yc",
    "outputId": "9e1badc9-a6c4-48b7-9125-e0810655528b"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bdcebc6a21ae41e3bb78834b4f244fae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00001-of-00004.safetensors:   0%|          | 0.00/4.98G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "89949427bf5142c29c54978c4f0ae52a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00002-of-00004.safetensors:   0%|          | 0.00/5.00G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a88b441b15714e138db6fa813dd82a47",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00003-of-00004.safetensors:   0%|          | 0.00/4.92G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1c4f8df93db246d18494820bb8ec37be",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00004-of-00004.safetensors:   0%|          | 0.00/1.17G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "combined_weights = {}\n",
    "\n",
    "for i in range(1, 5):\n",
    "    weights_file = hf_hub_download(\n",
    "        repo_id=\"meta-llama/Meta-Llama-3-8B-Instruct\",\n",
    "        filename=f\"model-0000{i}-of-00004.safetensors\",\n",
    "        local_dir=\"Llama-3-8B-Instruct\"\n",
    "    )\n",
    "    current_weights = load_file(weights_file)\n",
    "    combined_weights.update(current_weights)\n",
    "\n",
    "\n",
    "model = Llama3Model(LLAMA3_CONFIG_8B)\n",
    "load_weights_into_llama(model, LLAMA3_CONFIG_8B, combined_weights)\n",
    "model.to(device)\n",
    "del combined_weights  # free up memory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "VlH7qYVdDKQr",
   "metadata": {
    "id": "VlH7qYVdDKQr"
   },
   "source": [
    "- Note that the Llama 3 model should ideally be used with the correct prompt template that was used during finetuning (as discussed in chapter 7)\n",
    "- Below is a wrapper class around the tokenizer based on Meta AI's Llama 3-specific [ChatFormat code](https://github.com/meta-llama/llama3/blob/11817d47e1ba7a4959b025eb1ca308572e0e3963/llama/tokenizer.py#L202) that constructs the prompt template"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "4be5b481-1110-46e8-a931-3988d890cf8c",
   "metadata": {
    "id": "4be5b481-1110-46e8-a931-3988d890cf8c"
   },
   "outputs": [],
   "source": [
    "class ChatFormat:\n",
    "\n",
    "    def __init__(self, tokenizer: Tokenizer, *,\n",
    "                 default_system=\"You are a helpful assistant.\"):\n",
    "        self.tok = tokenizer\n",
    "        self.default_system = default_system\n",
    "\n",
    "    def _header(self, role):\n",
    "        \"\"\"Encode <|start_header_id|>role<|end_header_id|>\\n\\n\"\"\"\n",
    "        return (\n",
    "            [self.tok.special[\"<|start_header_id|>\"]]\n",
    "            + self.tok.encode(role)\n",
    "            + [self.tok.special[\"<|end_header_id|>\"]]\n",
    "            + self.tok.encode(\"\\n\\n\")\n",
    "        )\n",
    "\n",
    "    def encode(self, user_message, system_message=None):\n",
    "        sys_msg = system_message if system_message is not None else self.default_system\n",
    "\n",
    "        ids = [self.tok.special[\"<|begin_of_text|>\"]]\n",
    "\n",
    "        # system\n",
    "        ids += self._header(\"system\")\n",
    "        ids += self.tok.encode(sys_msg)\n",
    "        ids += [self.tok.special[\"<|eot_id|>\"]]\n",
    "\n",
    "        # user\n",
    "        ids += self._header(\"user\")\n",
    "        ids += self.tok.encode(user_message)\n",
    "        ids += [self.tok.special[\"<|eot_id|>\"]]\n",
    "\n",
    "        # assistant header (no content yet)\n",
    "        ids += self._header(\"assistant\")\n",
    "\n",
    "        return ids"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "M-dkSNvwDttN",
   "metadata": {
    "id": "M-dkSNvwDttN"
   },
   "source": [
    "- The usage is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "nwBrTGTsUNhn",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nwBrTGTsUNhn",
    "outputId": "72a495b4-b872-429a-88ef-49a9b4577f0f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[128000, 128006, 9125, 128007, 271, 2675, 527, 264, 11190, 18328, 13, 128009, 128006, 882, 128007, 271, 9906, 4435, 0, 128009, 128006, 78191, 128007, 271]\n"
     ]
    }
   ],
   "source": [
    "tokenizer = Tokenizer(tokenizer_file_path)\n",
    "chat_tokenizer = ChatFormat(tokenizer)\n",
    "\n",
    "token_ids = chat_tokenizer.encode(\"Hello World!\")\n",
    "print(token_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "0fpmpVgYVTRZ",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 36
    },
    "id": "0fpmpVgYVTRZ",
    "outputId": "bb3e819a-112a-466c-ac51-5d14a9c3475b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\nHello World!<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(token_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "Wo-aUGeKDvqq",
   "metadata": {
    "id": "Wo-aUGeKDvqq"
   },
   "source": [
    "- Let's now see the Llama 3 instruction model in action:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "ozGOBu6XOkEW",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ozGOBu6XOkEW",
    "outputId": "4f689c70-bed9-46f3-a52a-aea47b641283"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output text:\n",
      " Llamas are herbivores, which means they primarily eat plants and plant-based foods. Their diet typically consists of:\n",
      "\n",
      "1. Grasses: Llamas love to graze on grasses, including tall grasses, short grasses, and even weeds.\n",
      "2. Hay: Hay is a staple in a llama's diet. They enjoy a variety of hays, such as timothy hay, alfalfa hay, and oat hay.\n",
      "3. Grains: Llamas may be fed grains like oats, corn, and barley as a supplement to their diet.\n",
      "4. Fruits and vegetables: Llamas enjoy fruits and vegetables like apples, carrots, and sweet potatoes as treats or additions to their meals.\n",
      "5. Minerals:\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "token_ids = generate(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(\"What do llamas eat?\", chat_tokenizer).to(device),\n",
    "    max_new_tokens=150,\n",
    "    context_size=LLAMA3_CONFIG_8B[\"context_length\"],\n",
    "    top_k=1,\n",
    "    temperature=0.\n",
    ")\n",
    "\n",
    "output_text = token_ids_to_text(token_ids, tokenizer)\n",
    "\n",
    "\n",
    "def clean_text(text, header_end=\"assistant<|end_header_id|>\\n\\n\"):\n",
    "    # Find the index of the first occurrence of \"<|end_header_id|>\"\n",
    "    index = text.find(header_end)\n",
    "\n",
    "    if index != -1:\n",
    "        # Return the substring starting after \"<|end_header_id|>\"\n",
    "        return text[index + len(header_end):].strip()  # Strip removes leading/trailing whitespace\n",
    "    else:\n",
    "        # If the token is not found, return the original text\n",
    "        return text\n",
    "\n",
    "print(\"Output text:\\n\", clean_text(output_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2r5JKrO-ZOHK",
   "metadata": {
    "id": "2r5JKrO-ZOHK"
   },
   "source": [
    "&nbsp;\n",
    "# Llama 3.1 8B"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "QiQxX0XnP_iC",
   "metadata": {
    "id": "QiQxX0XnP_iC"
   },
   "source": [
    "- A few months after the initial Llama 3 release, Meta AI followed up with their Llama 3.1 suite of models (see the official [Introducing Llama 3.1: Our most capable models to date](https://ai.meta.com/blog/meta-llama-3-1/) announcement blog post for details)\n",
    "- Conveniently, we can reuse our previous Llama 3 code from above to implement Llama 3.1 8B\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gpt-to-llama/llama3-to-llama31.webp\" width=\"700px\">\n",
    "\n",
    "- The architecture is identical, with the only change being a rescaling of the RoPE frequencies as indicated in the configuration file below\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "X5Fg8XUHMv4M",
   "metadata": {
    "id": "X5Fg8XUHMv4M"
   },
   "outputs": [],
   "source": [
    "LLAMA3_CONFIG_8B = {\n",
    "    \"vocab_size\": 128_256,   # Vocabulary size\n",
    "    \"context_length\": 8192,  # Context length\n",
    "    \"emb_dim\": 4096,         # Embedding dimension\n",
    "    \"n_heads\": 32,           # Number of attention heads\n",
    "    \"n_layers\": 32,          # Number of layers\n",
    "    \"hidden_dim\": 14_336,    # Size of the intermediate dimension in FeedForward\n",
    "    \"n_kv_groups\": 8,        # Key-Value groups for grouped-query attention\n",
    "    \"rope_base\": 500_000.0,  # The base in RoPE's \"theta\"\n",
    "    \"rope_freq\": None,       # Additional configuration for adjusting the RoPE frequencies\n",
    "    \"dtype\": torch.bfloat16  # Lower-precision dtype to reduce memory usage\n",
    "}\n",
    "\n",
    "LLAMA31_CONFIG_8B = {\n",
    "    \"vocab_size\": 128_256,      # Vocabulary size\n",
    "    \"context_length\": 131_072,  # NEW: Larger supported context length\n",
    "    \"emb_dim\": 4096,            # Embedding dimension\n",
    "    \"n_heads\": 32,              # Number of attention heads\n",
    "    \"n_layers\": 32,             # Number of layers\n",
    "    \"hidden_dim\": 14_336,       # Size of the intermediate dimension in FeedForward\n",
    "    \"n_kv_groups\": 8,           # Key-Value groups for grouped-query attention\n",
    "    \"rope_base\": 500_000.0,     # The base in RoPE's \"theta\"\n",
    "    \"dtype\": torch.bfloat16,    # Lower-precision dtype to reduce memory usage\n",
    "    \"rope_freq\": {              # NEW: RoPE frequency scaling\n",
    "        \"factor\": 8.0,\n",
    "        \"low_freq_factor\": 1.0,\n",
    "        \"high_freq_factor\": 4.0,\n",
    "        \"original_context_length\": 8192,\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "xa3bpMDtTdBs",
   "metadata": {
    "id": "xa3bpMDtTdBs"
   },
   "source": [
    "- As we've seen in the code earlier, the RoPE method uses sinusoidal functions (sine and cosine) to embed positional information directly into the attention mechanism\n",
    "- In Llama 3.1, via the additional configuration, we introduce additional adjustments to the inverse frequency calculations\n",
    "- These adjustments influence how different frequency components contribute to the positional embeddings (a detailed explanation is a topic for another time)\n",
    "- Let's try out the Llama 3.1 model in practice; first, we clear out the old model to free up some GPU memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "7dUtYnNUOqhL",
   "metadata": {
    "id": "7dUtYnNUOqhL"
   },
   "outputs": [],
   "source": [
    "# free up memory\n",
    "del model\n",
    "\n",
    "gc.collect()  # Run Python garbage collector\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "DbbVsll6TYWR",
   "metadata": {
    "id": "DbbVsll6TYWR"
   },
   "source": [
    "- Next, we download the tokenizer\n",
    "- Note that since the Llama 3.1 family is distinct from the Llama 3 family, you'd have to go to the [meta-llama/Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B) repository and acknowledge the license terms for your Hugging Face access token to work for the download\n",
    "- Tip: For simplicity, we only load the base model below, but there's also an instruction-finetuned version you can use by replacing `\"meta-llama/Llama-3.1-8B\"` with `\"meta-llama/Llama-3.1-8B-Instruct\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "8xDk4chtPNU4",
   "metadata": {
    "id": "8xDk4chtPNU4"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ac808a4fe89d4ca89597a90f6ab83a30",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "original/tokenizer.model:   0%|          | 0.00/2.18M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tokenizer_file_path = hf_hub_download(\n",
    "    repo_id=\"meta-llama/Llama-3.1-8B\",\n",
    "    filename=\"original/tokenizer.model\",\n",
    "    local_dir=\"Llama-3.1-8B\"\n",
    ")\n",
    "\n",
    "tokenizer = Tokenizer(tokenizer_file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "a7l21VE4Otcs",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "a7l21VE4Otcs",
    "outputId": "3dd5cfba-bf3f-44d2-9be1-7cd42bfe4ba9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of parameters: 8,030,261,248\n"
     ]
    }
   ],
   "source": [
    "model = Llama3Model(LLAMA31_CONFIG_8B)\n",
    "\n",
    "total_params = sum(p.numel() for p in model.parameters())\n",
    "print(f\"Total number of parameters: {total_params:,}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "u4J7IxOvOyPM",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 145,
     "referenced_widgets": [
      "5bbaa046d8934c8fae0a12c3d7bd991b",
      "e1e4125eac004bae92dc1f22f673bf0e",
      "d5b4bb4891ec4e44be46e9815c7e10dc",
      "4f6595a392b244bd8e887935defc06f0",
      "100c1b15cc4046cea1147f657eb2d8d0",
      "81458e7953a349cfafccaa213b370406",
      "a3dc9dfadae642b4a873705596739468",
      "f55b59efcefa4ad5955d082f4bf7c637",
      "1b02e0c7d1604b1c87a327c4c4f8b0e7",
      "02ad170019454fd096b37347de5c481d",
      "c52e0f34892b4daa84c1bf61500ac399",
      "af985cf6fa26475eb2c4dd81e0c79ff4",
      "8659c3eddb014c3bb5931fd9e6fadad8",
      "f5fa00d96c4c49e48e1806d23a5b8570",
      "080c484114f64f5591fa1287a35b46c9",
      "14dc6a3717484c55a116612e28447dbb",
      "00d3286c9c1d4161bb777b7b65ae744d",
      "66f27fb11edf453b8144c2dfcdc66baa",
      "5798e5118430439fb1f6bf29e1bafe58",
      "357f367cf74146b8825be371acd51d06",
      "94073be250cd42d5b82e196e30cbf22e",
      "0cd0724f825e480389a82f0c49f91e6d",
      "dffa208978f34e6a9aae94ecda92fe67",
      "b8a98f163ebd4ac89af08a49c0881c23",
      "f0d9febe1a634a0ba7e8e50fa104dcc2",
      "e23870f0c7ff40cc8fa6a1e862a4af99",
      "87da9905a0534c26ad0712ad426ca930",
      "b953419300604b8e86fc0ad003fdfd2f",
      "f1865ed0fbcc40eeabdca90a43d00069",
      "ea0128909a9d4801ba312a876b0cf183",
      "d160986df978416c9ad91d1e10fc90fc",
      "5e97f7c2e8f5453dafcdad0552060e60",
      "4b3e7b8774df4b458bb6c6146fe3226d",
      "2ffd8dbed00e46d2887b9a2590cad297",
      "a06dcb3bdfc84905a7222066c32fe500",
      "e7602abc26714ee890a0cf5c0c7b67e1",
      "dc5d555099f64a998514ebde90eeb6df",
      "ef93a2f58cc54373941f43658bb808cf",
      "fea1e2327d2944859af3d91c216b9008",
      "320c00a5d18c45ccae634d166f1bd810",
      "6c857e69d5204cd3b7c3bf426993ad1f",
      "2145e47428f1446fba3e62b3cde0a7f5",
      "3d519ce3562c4e249bf392c7f43d04c0",
      "cc20ffcf0c1a4656945959bf457dfd84"
     ]
    },
    "id": "u4J7IxOvOyPM",
    "outputId": "925348d7-fc69-4d1b-90f1-7029426bcfcf"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4864b6a5f55340809e1e392cbeb5ca3c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00001-of-00004.safetensors:   0%|          | 0.00/4.98G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a7c77ab5f83a4319b66856b75cf04e1e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00002-of-00004.safetensors:   0%|          | 0.00/5.00G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e69661497025474b9523f5035634f788",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00003-of-00004.safetensors:   0%|          | 0.00/4.92G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f4ca91e917af4a37868e416717c9e762",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00004-of-00004.safetensors:   0%|          | 0.00/1.17G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "combined_weights = {}\n",
    "\n",
    "for i in range(1, 5):\n",
    "    weights_file = hf_hub_download(\n",
    "        repo_id=\"meta-llama/Llama-3.1-8B\",\n",
    "        filename=f\"model-0000{i}-of-00004.safetensors\",\n",
    "        local_dir=\"Llama-3.1-8B\"\n",
    "    )\n",
    "    current_weights = load_file(weights_file)\n",
    "    combined_weights.update(current_weights)\n",
    "\n",
    "load_weights_into_llama(model, LLAMA31_CONFIG_8B, combined_weights)\n",
    "model.to(device);\n",
    "del combined_weights  # free up memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "wJFnF8ATPbtD",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wJFnF8ATPbtD",
    "outputId": "67d5cb66-3588-4fd4-ac75-39bfe3aa82d8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output text:\n",
      " Every effort has been made to trace copyright holders and to obtain their permission for the use of copyright material. The publisher apologizes for any\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "token_ids = generate(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(\"Every effort\", tokenizer).to(device),\n",
    "    max_new_tokens=25,\n",
    "    context_size=LLAMA31_CONFIG_8B[\"context_length\"],\n",
    "    top_k=1,\n",
    "    temperature=0.\n",
    ")\n",
    "\n",
    "print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "DR9NBDUjPrDp",
   "metadata": {
    "id": "DR9NBDUjPrDp"
   },
   "source": [
    "&nbsp;\n",
    "# Llama 3.2 1B"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "imoxFiDzJcxk",
   "metadata": {
    "id": "imoxFiDzJcxk"
   },
   "source": [
    "- As of this writing, Meta AI's latest models are the Llama 3.2 models announced [here](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/)\n",
    "- The code for the Llama 3.2 text model is similar to that of Llama 3.1, except that the model has shrunk in size (there is a 1B and 3B version)\n",
    "- The other efficiency tweak was that they added back weight tying (a concept that was original used in the GPT-2 architecture); here, they reuse the same weight parameter values in the input (token) embedding layer and output layer\n",
    "- The small model size of Llama 3.2 1B is quite convenient, since it can even run on many mobile devices\n",
    "- The architectural differences between Llama 3.1 8B and Llama 3.2 1B are illustrated in the figure below"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "OL1EoXQ6TPb7",
   "metadata": {
    "id": "OL1EoXQ6TPb7"
   },
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/gpt-to-llama/llama31-to-llama32.webp?1\" width=\"700px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "K0KgjwCCJ9Fb",
   "metadata": {
    "id": "K0KgjwCCJ9Fb"
   },
   "source": [
    "- As we can see based on the figure above, the main difference between the Llama 3.1 8B and Llama 3.2 1B architectures are the respective sizes\n",
    "- A small additional change is an increased RoPE rescaling factor, which is reflected in the configuration file below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "Yv_yF3NCQTBx",
   "metadata": {
    "id": "Yv_yF3NCQTBx"
   },
   "outputs": [],
   "source": [
    "LLAMA31_CONFIG_8B = {\n",
    "    \"vocab_size\": 128_256,      # Vocabulary size\n",
    "    \"context_length\": 131_072,  # NEW: Larger supported context length\n",
    "    \"emb_dim\": 4096,            # Embedding dimension\n",
    "    \"n_heads\": 32,              # Number of attention heads\n",
    "    \"n_layers\": 32,             # Number of layers\n",
    "    \"hidden_dim\": 14_336,       # Size of the intermediate dimension in FeedForward\n",
    "    \"n_kv_groups\": 8,           # Key-Value groups for grouped-query attention\n",
    "    \"rope_base\": 500_000.0,     # The base in RoPE's \"theta\"\n",
    "    \"dtype\": torch.bfloat16,    # Lower-precision dtype to reduce memory usagey\n",
    "    \"rope_freq\": {              # NEW: RoPE frequency scaling\n",
    "        \"factor\": 8.0,\n",
    "        \"low_freq_factor\": 1.0,\n",
    "        \"high_freq_factor\": 4.0,\n",
    "        \"original_context_length\": 8192,\n",
    "    }\n",
    "}\n",
    "\n",
    "\n",
    "LLAMA32_CONFIG_1B = {\n",
    "    \"vocab_size\": 128_256,      # Vocabulary size\n",
    "    \"context_length\": 131_072,  # Context length\n",
    "    \"emb_dim\": 2048,            # NEW: Half the embedding dimension\n",
    "    \"n_heads\": 32,              # Number of attention heads\n",
    "    \"n_layers\": 16,             # NEW: Half the number of layers\n",
    "    \"hidden_dim\": 8192,         # NEW: Almost half the size of the intermediate dimension in FeedForward\n",
    "    \"n_kv_groups\": 8,           # Key-Value groups for grouped-query attention\n",
    "    \"rope_base\": 500_000.0,     # The base in RoPE's \"theta\"\n",
    "    \"dtype\": torch.bfloat16,    # Lower-precision dtype to reduce memory usage\n",
    "    \"rope_freq\": {              # RoPE frequency scaling\n",
    "        \"factor\": 32.0,         # NEW: Adjustment of the rescaling factor\n",
    "        \"low_freq_factor\": 1.0,\n",
    "        \"high_freq_factor\": 4.0,\n",
    "        \"original_context_length\": 8192,\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "Dl4_0EoJKKYv",
   "metadata": {
    "id": "Dl4_0EoJKKYv"
   },
   "source": [
    "- Below, we can reuse the code from the Llama 3.1 8B section to load the Llama 3.2 1B model\n",
    "- Again, since the Llama 3.2 family is distinct from the Llama 3.1 family, you'd have to go to the [meta-llama/Llama-3.2-1B](https://huggingface.co/meta-llama/Llama-3.2-1B) repository and acknowledge the license terms for your Hugging Face access token to work for the download\n",
    "- Tip: For simplicity, we only load the base model below, but there's also an instruction-finetuned version you can use by replacing `\"meta-llama/Llama-3.2-1B\"` with `\"meta-llama/Llama-3.2-1B-Instruct\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "tCstHgyRRD2x",
   "metadata": {
    "id": "tCstHgyRRD2x"
   },
   "outputs": [],
   "source": [
    "# free up memory\n",
    "del model\n",
    "\n",
    "\n",
    "gc.collect()  # Run Python garbage collector\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "jt8BKAHXRCPI",
   "metadata": {
    "id": "jt8BKAHXRCPI"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7658da8b2a5e4273b45c35411bdba8a0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "original/tokenizer.model:   0%|          | 0.00/2.18M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tokenizer_file_path = hf_hub_download(\n",
    "    repo_id=\"meta-llama/Llama-3.2-1B\",\n",
    "    filename=\"original/tokenizer.model\",\n",
    "    local_dir=\"Llama-3.2-1B\"\n",
    ")\n",
    "\n",
    "tokenizer = Tokenizer(tokenizer_file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "uf8KjasmRFSt",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uf8KjasmRFSt",
    "outputId": "4e718852-2aa1-4b5a-bec3-3d5f866a4038"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of parameters: 1,498,482,688\n",
      "\n",
      "Total number of unique parameters: 1,235,814,400\n"
     ]
    }
   ],
   "source": [
    "model = Llama3Model(LLAMA32_CONFIG_1B)\n",
    "\n",
    "total_params = sum(p.numel() for p in model.parameters())\n",
    "print(f\"Total number of parameters: {total_params:,}\")\n",
    "\n",
    "# Account for weight tying\n",
    "total_params_normalized = total_params - model.tok_emb.weight.numel()\n",
    "print(f\"\\nTotal number of unique parameters: {total_params_normalized:,}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc004791-9e28-4872-9ae9-fb51c6c83d7c",
   "metadata": {},
   "source": [
    "- Alternatively, we can use more robust function that factors in the weight tying based on shared data pointers in memory as suggested in [#822](https://github.com/rasbt/LLMs-from-scratch/issues/822):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "7aaeb28e-62ab-4711-9f07-1b32ac9dbeba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Total number of unique parameters: 1,498,482,688\n"
     ]
    }
   ],
   "source": [
    "def count_unique_parameters(model):\n",
    "    unique_params = set()\n",
    "    total_unique_params = 0\n",
    "    \n",
    "    for param in model.parameters():\n",
    "        if param.data_ptr() not in unique_params:\n",
    "            total_unique_params += param.numel()\n",
    "            unique_params.add(param.data_ptr())\n",
    "            \n",
    "    return total_unique_params\n",
    "\n",
    "total_params_uniq = count_unique_parameters(model)\n",
    "print(f\"\\nTotal number of unique parameters: {total_params_uniq:,}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "9FbCIYW7RIOe",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9FbCIYW7RIOe",
    "outputId": "35588405-e2e1-4871-a1db-1d4bcb852e49"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ebf98f844b6b49669d51601cbceea91e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/2.47G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model uses weight tying.\n"
     ]
    }
   ],
   "source": [
    "weights_file = hf_hub_download(\n",
    "    repo_id=\"meta-llama/Llama-3.2-1B\",\n",
    "    filename=\"model.safetensors\",\n",
    "    local_dir=\"Llama-3.2-1B\"\n",
    ")\n",
    "current_weights = load_file(weights_file)\n",
    "\n",
    "load_weights_into_llama(model, LLAMA32_CONFIG_1B, current_weights)\n",
    "model.to(device);\n",
    "del current_weights  # free up memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "pPp5yjir6FYJ",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pPp5yjir6FYJ",
    "outputId": "6c8e79d2-0769-43a7-93b3-f04c030e1aac"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weight tying: True\n"
     ]
    }
   ],
   "source": [
    "# Checks that the weight values are the same\n",
    "print(\"Weight tying:\", torch.equal(model.tok_emb.weight, model.out_head.weight))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "b2bdebe0-d2b0-4d33-8b7e-1b4f9a02ca12",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weight tying: True\n"
     ]
    }
   ],
   "source": [
    "# Furthermore, check if PyTorch uses the same underlying memory\n",
    "print(\"Weight tying:\", model.tok_emb.weight.data_ptr() == model.out_head.weight.data_ptr())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "3kh7yrw2W4qr",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3kh7yrw2W4qr",
    "outputId": "b7e66a17-57ec-4b0e-c4ff-8d9a6b8e6ea5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output text:\n",
      " Every effort is made to ensure that the information on this website is accurate and up to date. However, the information is provided without any\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "token_ids = generate(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(\"Every effort\", tokenizer).to(device),\n",
    "    max_new_tokens=25,\n",
    "    context_size=LLAMA32_CONFIG_1B[\"context_length\"],\n",
    "    top_k=1,\n",
    "    temperature=0.\n",
    ")\n",
    "\n",
    "print(\"Output text:\\n\", token_ids_to_text(token_ids, tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "VO4Qf0zyW1ZC",
   "metadata": {
    "id": "VO4Qf0zyW1ZC"
   },
   "source": [
    "&nbsp;\n",
    "# What's next?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "CjCewpo2XPAd",
   "metadata": {
    "id": "CjCewpo2XPAd"
   },
   "source": [
    "- This notebook concludes the conversion from GPT to Llama 3.2\n",
    "- If you are interested in a more compact, standalone notebook, which only contains the Llama 3.2 code, check out the [standalone-llama32.ipynb](standalone-llama32.ipynb) notebook"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "A100",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
